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Foreword 


This  study  is  about  system-level  experimentation  (SLE)  -  campaigns  of  genuine 
discovery  experiments  -  and  how  SLEs  can  be  used  to  drive  disruptive  innovation. 

SLEs  support  the  discovery,  exploration,  and  understanding  of  new  operational  and 
system  concepts  and  the  technology  needed  to  support  those  concepts  in  future  environments. 
As  such,  SLEs  are  complimentary  to,  but  different  from  ATDs  or  JEFX  (focus  on  early 
discovery  versus  later  demonstration),  Battlelabs  (focus  on  game  changers  versus  near-term 
needs),  or  Big  Safari  or  the  Rapid  Capabilities  Office  (focus  on  discovery  in  future  environments 
versus  fast  fielding  to  meet  high  priority  needs).  Similarly,  SLEs  are  not  wargames  (e.g.,  the 
AF/A8  Futures  Game),  but  SLEs  could  be  conducted  in  the  futuristic  scenarios  of  such  games. 

Disruptive  innovations  often  arise  from  the  “friction  of  war.”  To  mimic  that  effect,  SLEs 
must  incorporate  a  challenge-competitive  environment  to  maximize  depth  of  innovation  and 
exploration,  with  an  unfettered,  highly  skilled  adversary  with  no  cultural  limitations  and  with 
technical  restrictions  imposed  only  by  physics.  The  experiments  should  be  staffed  with  carefully 
selected  individuals  who  have  attributes  conducive  to  “out  of  the  box”  exploration.  Experiments 
can  be  conducted  in  gaming  environments  ranging  from  simple  “seminar  explorations”  to 
networked  gaming  to  being  in  the  field.  The  SLE  approach  developed  in  this  study  also 
integrates  recently  codified  industry  innovation  practices  (e.g.,  innovation  starts  with  the  CEO). 


The  study  identifies  four  essential  components  in  the  development  of  disruptive 
innovation  by  means  of  system-level  experimentation:  ideas,  people,  venue,  and  experiments. 


•  Ideas:  Innovation  occurs  throughout  an  organization  and  must  be  sought  out.  It  is 
critical  to  identify  ideas  that  challenge  standard  ways  of  doing  things. 

•  People:  Not  all  people  are  innovative.  Those  that  are  must  be  identified,  supported, 
protected,  and  valued. 

•  Venue:  A  venue  is  not  a  specific  place  or  facility.  It  is  an  exploration  space,  which 
might  be  a  virtual  environment  or  the  battlefield  of  a  war  game. 

•  Experiments:  The  only  way  to  explore  the  complexities  of  a  system  is  through 
campaigns  of  experiments,  based  on  the  proper  venue,  people,  and  ideas.  Combining 
these  into  a  rigorous  program  of  technology  and  CONOPS  will  create  a  deep 
understanding  of  what  the  future  may  be  and  how  to  best  meet  it. 


A  case  study  based  on  the  development  of  the  armed  Predator  is  used  to  illustrate  the  SLE 
approach.  The  single  recommendation  is  to  replicate  this  case  study  through  the  auspices  of  a 
Chief  of  Innovation  reporting  to  the  CSAF.  The  recent  creation  of  an  Air  Force  Futures  Program 
between  AF/A8  and  AU/CC  provides  a  near-term  means  to  implement  this  recommendation 
without  creating  a  new  organization.  Over  time,  the  Air  Force  might  consider  creating  a  civilian 
IPA  position  for  someone  who  served  as  a  Chief  of  Innovation  in  industry. 


Stephen  E.  Cross  Scott  Fouse 

Co-Chairman  Co-Chairman 

System-Level  Experimentation  Study  System-Level  Experimentation  Study 


Public  Release 
Hi 


Public  Release 


(This  Page  Intentionally  Left  Blank.) 


Public  Release 
iv 


Public  Release 


Table  of  Contents 

Foreword . iii 

Table  of  Contents . v 

Executive  Summary . 1 

Annotated  Brief. . 15 

Appendix  A:  Terms  of  Reference . 39 

Appendix  B:  Study  Members . 41 

Appendix  C:  Visits  and  Briefings . 43 

Appendix  D:  Acronyms  and  Abbreviations . 45 

Appendix  E:  Case  Histories  in  Disruptive  Innovation . 47 

Appendix  F:  References . 59 

Appendix  G:  Initial  Distribution . 63 


Public  Release 


Public  Release 


(This  Page  Intentionally  Left  Blank.) 


Public  Release 


Public  Release 


Executive  Summary 


Introduction 

...  any  air  force  which  does  not  keep  its  doctrines  ahead  of  its  equipment,  and  its  vision  far 
into  the  future,  can  only  delude  the  nation  into  a  false  sense  of  security. 

General  Henry  “Hap”  Arnold 
November  1945 

Sixty-one  years  ago  General  Arnold,  as  the  Chief  of  the  Army  Air  Forces,  recognized  the 
importance  of  understanding  technology  and  tactics  required  to  dominate  in  future  environments. 
In  today’s  parlance,  General  Arnold  was  talking  about  maintaining  the  Air  Force’s  combat  edge 
through  innovation  (e.g.,  pursuing  revolutionary,  game-changing  systems).  In  the  commercial 
world,  innovation  is  a  means  to  stay  competitive;  in  the  military  world  it  is  a  means  for  achieving 
and  maintaining  dominance. 

This  is  a  Quick  Look  report  on  System-Level  Experimentation  (SLE).  Quick  Look 
studies  are  conducted  by  a  smaller  study  team  in  a  shorter  period  of  time  than  a  full  Scientific 
Advisory  Board  (SAB)  study.  This  study  is  about  provoking  disruptive  innovations  through 
system-level  experimentation.  There  are  five  key  points  in  this  study. 

1 .  SLE  is  an  approach  to  unfettered  exploration  of  alternative  system  concepts  focused 
in  future  environments  to  aid  in  earlier  discovery  of  game  changing  ways  to  “fly  and 
fight.” 

2.  Our  adversaries  innovate  while  we  merely  react  -  a  more  proactive  approach  would 
support  system-level  discovery  experiments  focused  on  disruptive  ideas,  thus 
increasing  the  likelihood  of  finding  game-changing  ways  to  “fly  and  fight”  BEFORE 
the  fight,  forcing  our  adversaries  to  react  to  our  innovations. 

3.  At  one  time  the  USAF  aggressively  searched  for  “game  changers;”  arguably  it  has  not 
done  so  for  many  years. 

4.  The  USAF  is  not  alone.  Innovation  is  also  an  important  national  issue  for  U.S. 
industry.  There  has  been  impressive  progress  in  the  last  year,  and  there  are  lessons 
learned  the  USAF  can  apply  now. 

5.  Ad  hoc  groups  (e.g.,  “innovation  task  forces”)  drawn  from  Air  Force  organizations 
can  be  fonned  to  do  SLEs  and  would  need  to  be  supported  by  a  small  group  of 
facilitators.  These  "innovation  task  forces"  need  to  be  resourced,  protected,  and 
visibly  important  to  the  Chief  of  Staff. 

The  Air  Force  faces  a  world  rapidly  increasing  in  complexity,  with  adversaries  ranging 
from  emerging  super  powers  to  elusive  terrorist  organizations.  With  the  quickening  pace  of 
technological  change  and  its  commercial  availability,  as  well  as  the  ability  of  our  adversaries  to 
take  advantage  of  that  technology  in  innovative  ways,  the  Air  Force  can  no  longer  solely  rely  on 
its  current  approach  to  developing  new  technologies  and  tactics.  The  Air  Force  must  become 
more  adaptable  and  able  to  respond  rapidly  to  opportunities  afforded  by  technological  advances. 
Indeed,  the  Air  Force  must  take  a  leadership  role  in  driving  the  development  of  new  technology 

Public  Release 
1 


Public  Release 


and  in  creating  the  tactics,  techniques,  and  procedures  to  use  that  technology.  In  short,  the  Air 
Force  must  become  more  innovative. 

Technology  and  tactics  are  inseparable,  so  that  the  Air  Force  must  embrace  an  approach 
for  innovation  that  links  technologists  and  operators  from  its  inception.  The  Air  Force  also  needs 
to  focus  on  an  uncertain  future,  with  unforeseen  requirements.  To  best  meet  that  future,  the  Air 
Force  must  embrace  a  culture  of  experimentation,  in  which  technology  and  tactics  are  developed 
and  tested  in  a  diverse  range  of  environments  -  the  experiments  should  be  focused  at  the  system 
level.  Experimentation  alone  is,  however,  not  sufficient.  The  Air  Force  must  also  develop  a 
strong  culture  that  fosters  and  nourishes  innovation.  This  quick  look  study  examined  how  the 
Air  Force  can  employ  system-level  experimentation  to  help  provoke  the  innovation  needed  to 
meet  its  future  challenges. 

Experimentation 

This  study  is  about  system-level  experimentation  and  how  it  drives  innovation.  By 
system  we  mean  the  integration  of  technology,  process,  people,  and  organization  to  accomplish 
some  set  of  functions.  In  the  context  of  this  study,  a  system  is  the  combination  of  the 
components  necessary  to  carry  out  the  missions  of  the  Air  Force. 

Alberts  and  Hayes  [1]  describe  three  types  of  experimentation: 

Discover v  experiments  are  designed  to  generate  new  ideas  or  ways  of  doing  things. 
Hypothesis  testing  experiments  are  the  classic  type  used  by  scholars  and  researchers  to 
advance  knowledge  by  seeking  to  falsify  specific  if/then  statements  or  to  discover  their 
limiting  conditions.  Demonstration  experiments  create  a  venue  in  which  known  truth  is 
recreated. 

Most  Air  Force  “experiments”  are  demonstrations  in  that  they  are  used  to  train  or  show 
that  expected  results  are  true.  For  the  purposes  of  this  study,  the  focus  is  only  on  discovery  and 
hypothesis  testing  experiments. 

Campaigns  of  experiments  are  groups  of  experiments  designed  for  a  given  purpose.  In  a 
typical  campaign,  there  are  a  series  of  coordinated  experiments,  with  each  experiment  informing 
the  next.  Such  campaigns  iterate  to  an  outcome  or  explore  a  wide  range  of  alternatives,  as  will 
be  discussed  in  more  detail  below. 

System-level  experimentation  involves  campaigns  of  discovery  experiments  that  combine 
technology  (both  developed  and  envisioned)  and  concept  of  operations  (CONOPS)  to  develop 
new  capabilities.  SLEs  explore  a  wide  range  of  possible  future  environments  and  conditions. 


Innovation 

Clayton  Christensen,  in  his  1997  book  The  Innovator's  Dilemma.  [2]  introduces  the 
concept  of  a  disruptive  technology  to  describe  a  new  technological  innovation,  product,  or 
service  that  eventually  overturns  the  dominant  technology  or  product  existing  in  the  market.  He 
broadened  that  concept  to  disruptive  innovation  in  his  sequel  The  Innovator’s  Solution,  [3]  where 
he  notes  that  few  technologies  are  intrinsically  disruptive;  it  is  their  use  that  creates  their 
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disruptive  impact.  He  describes  sustaining  innovation,  on  the  other  hand,  as  successive 
incremental  improvements  to  perfonnance  that  are  incorporated  into  existing  products. 

Sustaining  innovation  is  the  most  common  form  of  innovation.  One  example  is  the 
automobile,  which  today  is  only  different  in  degree  from  Henry  Ford’s  Model  T.  The  personal 
computer  is  faster  with  more  memory  and  disk  space  today  than  in  the  early  1980s,  but  its 
functionality  is  no  different.  Most  Air  Force  technologies  under  current  development  fall  into 
the  category  of  “sustaining  innovation.” 

Disruptive  innovation  arises  not  just  from  the  introduction  of  a  new  technology,  but  also 
through  changes  in  the  use  of  those  technologies.  Disruptive  innovations  often  impact  the 
market  broadly,  creating  entirely  new  industries  while  destroying  others,  when  first  introduced. 
The  automobile  changed  the  world  by  eliminating  horses  as  the  principle  mode  of  transportation, 
with  similar  impacts  in  all  the  subsidiary  businesses  associated  with  them.  In  the  late  1970s, 
computing  changed  in  a  disruptive  manner  with  the  introduction  of  the  personal  computer.  At 
first  PCs  were  slow,  had  no  memory  or  software,  and  their  utility  for  almost  anything  was 
questioned.  Their  performance  was  far  less  than  the  mainframe  computers  they  eventually 
replaced.  They  were,  however,  relatively  inexpensive  and,  more  importantly,  could  be  run  and 
managed  by  an  individual,  not  requiring  an  extensive  organization.  Suddenly,  computing  was 
put  into  the  hands  of  almost  everyone,  which  led  to  extraordinary  changes  in  the  computing 
business  as  well  as  in  almost  all  other  businesses  and,  essentially,  every  aspect  of  life.  The 
disruption  arises  from  the  use  of  the  technology,  which  changed  computing  in  a  drastic  way. 
Coupling  personal  computing  with  the  internet  is  leading  to  a  revolution  in  connectivity,  with 
blogs,  podcasts,  wikis,  and  etc.,  changing  how  the  world  communicates.  The  combination  of 
technology  and  use  (CONOPS)  form  the  basis  of  this  study. 


Disruptive  innovation  in  warfare 

Carl  von  Clausewitz,  a  Prussian  general  and  military  theorist,  introduced  the  idea  of  the 
“friction  of  war”  to  describe  how  war  in  reality  differs  from  that  on  paper  -  things  that  can  go 
wrong  often  do  go  wrong.  The  friction  of  competition  has  led  to  many  disruptive  innovations  in 
which  new  ways  of  using  technology  to  advance  CONOPS  were  developed  and  fielded.  Just  a 
few  examples  from  American  aviation  are  listed.  However,  this  study  chose  to  highlight  two 
examples  from  the  great  Air  Force  innovators  of  the  past  as  well  some  more  modern  examples. 

After  World  War  I  ended,  large  battleships  were  believed  to  be  invincible,  and  thus 
garnered  a  major  part  of  Defense  funds.  General  Billy  Mitchell  felt  that  no  naval  fleet  could 
survive  an  attack  by  land-based  air  and,  after  an  extensive  and  bitter  controversy,  forced  an 
experiment  with  decommissioned  U.S.  capital  ships  and  the  “unshakable”  captured  Ostfriedland 
as  targets.  Bombers  sank  them  all,  proving  his  ideas.  This  “experiment”  made  the  Navy  realize 
that  a  future  force  based  solely  on  the  battleship  was  too  risky,  so  it  shifted  its  focus  from 
battleships  to  aircraft  carriers,  combining  air  and  sea  power  in  a  way  that  changed  naval  warfare 
in  a  disruptive  way. 

In  the  1930s,  standard  Air  Corps  doctrine  assumed  that  pursuit  planes  would  be 
ineffective  against  bombers;  the  war  games  of  the  day  supported  that  view.  Captain  Claire  Lee 
Chennault,  however,  felt  the  Air  Corps  rigged  maneuvers  in  favor  of  bombers.  To  prove  his 
point,  Captain  Chennault  arranged  for  a  test  of  his  ideas  by  setting  an  early  warning  system  in  an 
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exercise  held  in  1933.  He  placed  spotters  in  the  region  of  the  test  with  telephone  communication 
back  to  the  fighter  bases.  When  bombers  were  spotted,  the  fighters  scrambled  and  intercepted 
the  bombers.  Despite  protests  by  the  bombers,  who  felt  the  exercise  was  “unfair,”  Chennault’s 
experiment  clearly  showed  that  pursuit  planes  could  intercept  bombers.  Unfortunately,  the  Air 
Corps  largely  ignored  these  results  at  that  time.  However,  the  victory  of  pursuit  planes  in  the 
Battle  of  Britain  verified  his  ideas,  leading  to  a  new  age  of  air  warfare. 

There  are  a  number  of  examples  of  disruptive  innovations  from  more  recent  eras  as  well. 
While  we  describe  them  by  their  technology,  the  disruption  arose  from  the  change  in  doctrine 
enabled  by  the  technologies;  they  changed  how  the  Air  Force  did  its  job.  Some  examples  are  the 
combining  of  technologies  to  create  the  Intercontinental  Ballistic  Missile  (ICBM),  the 
development  of  precision-guided  munitions,  stealth  technology,  etc.  An  interesting  example 
occurred  in  2000,  when  the  CSAF  ordered  a  rapid  study  (90  days)  to  see  if  a  UAV  (the  Predator) 
could  be  armed  with  Hellfire  missiles.  The  CSAF  asked  the  technologists  and  operators  to  assess 
feasibility  and  tactics.  He  allocated  resources  and  gave  them  the  right  to  fail.  The  disruptive 
innovation  was  not,  however,  the  addition  of  a  missile  to  an  unmanned  platform.  The  innovation 
came  about  when  the  system  was  put  into  the  field  (the  “experiment”)  and  changed  an 
observation  platform  into  an  offensive  weapon.  How  disruptive  this  innovation  will  be  is  still 
unknown. 

All  of  these  examples  show  that  disruptive  innovation  is  much  more  than  technology.  It 
is  the  combination  of  technology  and  CONOPS,  often  developed  through  experimentation,  that 
have  led  to  fundamental  shifts  in  how  warfare  is  conducted.  The  study  team  notes  that  none  of 
these  examples  resulted  from  the  formal  requirements  process  and,  in  fact,  threatened  to  disrupt 
the  plans  that  are  pursued  by  the  formal  requirements  and  acquisition  process. 


The  Air  Force  today 

This  quick  look  study  could  not  research  every  Air  Force  organization  nor  could  it 
examine  all  research  and  development  activities.  The  study  team  did,  however,  interact  with  a 
wide  range  of  organizations  throughout  the  Air  Force  and  thus  feels  comfortable  with  the 
assessment  of  the  current  state  of  Air  Force  practices.  The  team  has  the  same  position  on  the 
culture  of  the  Army  and  Navy  as  well. 

The  Air  Force  is  very  good  at  sustaining  innovation.  The  Air  Force  continues  to  advance 
technology  throughout  the  service.  However,  it  has  often  been  a  challenge  to  deploy  those 
advances  in  a  timely  way  due  to  the  well-known  issues  with  the  acquisition  process.  Especially 
when  faced  with  an  innovative  adversary  (e.g.,  IEDs),  the  cumbersome  processes  limit  the  ability 
to  respond  in  an  effective  way.  We  note,  however,  that  recent  acquisition  improvements  are 
targeting  this  issue  by  focusing  on  processes  to  more  rapidly  field  technologies. 

The  Air  Force  lacks  a  focus  on  true  discovery  experiments .  Since  the  development  of 
stealth  technology  and  accompanying  operational  procedures,  the  Air  Force  has  not  conducted 
major  discovery  experiments  at  the  systems  level.  JEFX  organizers  have  primarily  devoted  their 
exercises  to  evaluating  “leave  behind”  capabilities  and  conducting  limited  demonstrations  with  a 
high  probability  for  success.  The  original  intent  to  explore  alternative  concepts  of  operation 
enabled  by  modem  technology  has  acquiesced  to  concern  over  near-tenn  needs  and  budgets. 
Indeed,  the  reliance  on  demonstrations  rather  than  experiments  seems  pervasive  throughout  the 
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Air  Force.  Organizers  even  constrain  adversarial  experiments  to  fit  training  scenarios,  with  red 
teams  forced  to  follow  scripts  that  prevent  them  from  expressing  creative,  disruptive  responses  to 
situations.  Indeed,  the  team  even  heard  from  one  experimenter  that  a  competition  was  set  up  so 
that  the  blue  team  would  “win”  so  they  would  not  “feel  bad.” 

The  Air  Force  has  innovative  people,  yet  it  fails  to  take  full  advantage  of  them  and, 
indeed,  sometimes  hinders  them.  Over  the  course  of  the  study  the  team  met  many  creative 
Ainnen  (both  military  and  civilian)  predisposed  to  the  types  of  disruptive  thinking  fundamental 
to  this  study.  They  uniformly  expressed  frustration  at  the  lack  of  a  forum  to  present  their  ideas 
and  at  the  impediments  for  advancing  those  ideas  as  system  solutions  fielded  for  the  war  fighter. 
Both  the  Air  Force’s  bureaucratic  acquisition  process  and  risk-adverse  culture  limit  their 
effectiveness  as  innovators.  A  critical  need,  for  example,  in  any  innovative  organization  is  the 
free  flow  of  information.  One  of  the  most  disappointing  findings  of  the  study  was  the 
identification  of  intentional  efforts  to  stifle  information  exchange  around  the  USAF.  Borne  of 
fears  over  cyber-security,  the  USAF  communications  systems  managers  implement  an 
information  lock-down  that  too  often  defies  logic.  No  one  questions  the  worth  of  readiness  for 
cyber  attacks  and  network  warfare,  but  restrictive  security  practices  frequently  hamper 
reasonable,  necessary  information  exchange.  The  Air  Force  needs  policies  and  technologies  that 
balance  network  security  and  the  need  to  collaborate  effectively  to  pursue  innovation. 

The  Air  Force  has  largely  lost  its  ability  to  foster  disruptive  innovation.  The  future 
demands  that  the  Air  Force  once  again  become  a  leader  in  disruptive  innovation,  and  the  rest  of 
this  report  outlines  our  suggestions  initiating  this  transition. 

Lessons  Learned  from  the  Commercial  Sector 

Over  the  past  few  years  there  have  been  numerous  studies  on  the  status  of  innovation  in 
industry  [4,5,6].  From  these  studies,  a  common  view  of  what  it  takes  to  be  an  innovative 
company  is  emerging. 

1)  The  CEO  needs  to  be  the  “ owner  ”  of  innovation.  In  the  private  sector  it  is 
increasingly  understood  that  the  CEO  is  responsible  for  creating  a  culture  of  innovation. 
“Leading,  setting  direction,  laying  the  cultural  groundwork  that  stimulates  innovation  -  it’s 
essential  work  for  a  CEO,”  acknowledged  one  executive  [4].  This  does  not  mean  that  the  CEO  is 
the  principal  innovator.  Rather,  they  are  responsible  for  establishing  the  practices,  organizations, 
and  communication  channels  that  create  a  culture  in  which  innovation  can  flourish. 

2)  Chief  of  Innovation.  Successful  CEOs  often  appoint  a  single  person  who  is 
responsible  for  innovation,  the  “Idea  Tsar.”  That  chief  innovator  is  not,  in  general,  the  General 
Manager  of  R&D.  Rather  it  is  someone  who  can  balance  emerging  technology  with  the 
possibilities  of  the  marketplace.  This  is  a  function  that  demands  experience,  someone  who  has 
experienced  the  effect  of  a  disruptive  innovation  and  who  understands  the  difficulty  in  predicting 
how  a  given  technology  will  be  fully  utilized. 

3)  Organization.  Large  organizations  often  become  hierarchical,  which  tends  to  stifle 
the  communication  necessary  for  innovation.  A  successful  way  around  this  problem  is  to  create 
small  passion-driven  teams  to  focus  strictly  on  innovation  (e.g.,  Whirlpool,  Kodak,  etc.).  These 
teams  are  separate  from  the  main  lines  of  the  organization  and  are  thus  freed  from  at  least  some 
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of  the  standard  organization  constraints.  They  also  have  the  luxury  of  bringing  together  people 
that  may  have  been  separated  due  to  organizational  boundaries. 

4)  Ideas.  Innovation  cannot  happen  without  ideas,  and  these  ideas  can  come  from  any 
part  of  an  organization  as  well  as  outside  the  organization.  An  organization  successful  at 
innovation  must  actively  stimulate  the  generation  of  new  ideas,  seek  out  those  ideas,  explore 
competing  ones,  and  ensure  that  communication  flows  freely  so  that  “cross  pollination”  can 
occur.  Creativity  is  a  non-linear  process;  the  total  is  often  much  more  than  the  sum  of  its  parts. 
The  key  is  to  listen  to  all  voices  in  the  search  for  game-changing  ideas. 

5)  People.  Not  everyone  is  innovative.  It  is  important  to  identify  innovative  people  and 
to  protect  them  from  a  system  that  rewards  concrete  results  tied  to  a  mission.  Since  ideas 
comprise  an  innovator’s  primary  “product”  and  a  high  percentage  of  ideas  never  reach  fruition, 
management  must  not  punish  innovators  for  “bad”  ideas;  they  should  evaluate  innovators  on 
their  idea-generation  processes  and  reward  them  for  sound  decision  making  and  thorough 
investigation  of  a  topic.  Often  the  good  ideas  and  good  people  require  a  champion  to  shepherd 
them  through  the  organization. 

6)  Patience.  Innovation  is  a  dynamic  process  and  not  without  its  fits  and  starts.  It  is  a 
process  that  may  not  lead  immediately  to  an  improved  product  or  strategy.  It  is  important  to 
experiment  with  new  ideas  and  to  fail  early  and  often  to  find  what  ideas  actually  work.  This  is 
probably  the  most  difficult  element  to  deal  with,  since  there  is  so  much  pressure  for  results  now, 
but  the  reality  is  that  patience  can  allow  the  creative  process  to  play  out  and  then  to  gain  the 
insights  that  will  be  the  basis  for  the  revolutionary  game-changing  ideas. 


A  Framework  for  System-Level  Experimentation 

As  previously  noted,  innovation  often  arises  from  the  “friction  of  war.”  To  mimic  that 
effect,  SLEs  must  incorporate  a  challenge-competitive  environment  to  maximize  depth  of 
innovation  and  exploration,  with  an  unfettered,  highly  skilled  adversary  with  no  cultural 
limitations  and  with  technical  restrictions  imposed  only  by  physics.  As  suggested,  the 
experiments  should  be  staffed  with  carefully  selected  individuals  who  have  attributes  conducive 
to  “out  of  the  box”  exploration.  Experiments  can  be  conducted  in  gaming  environments  ranging 
from  simple  “seminar  explorations”  to  networked  games  or  even  exercises  in  the  field.  To 
achieve  a  full  exploration  of  possible  solution  space  requires  a  sustained  campaign  of  game- 
based  experiments,  capturing,  archiving,  and  mining  the  results  to  find  patterns  and  insights  that 
guide  further  system-level  iterations  and  future  experiments.  The  SLE  process  will  result  in  a 
shared  understanding  of  how  technology  and  operations  mesh  together  to  form  new  capabilities 
for  the  future. 

Through  the  analysis  of  the  commercial  lessons  learned,  we  identified  four  essential 
components  in  the  development  of  disruptive  innovation  by  means  of  system-level 
experimentation:  ideas,  people,  venue,  and  experiments.  Before  describing  the  components  in 
detail,  we  can  sum  up  the  basic  attributes  for  system-level  experimentation  as: 

•  Ideas:  Innovation  occurs  throughout  an  organization  and  must  be  sought  out.  It  is 
critical  to  identify  ideas  that  challenge  standard  ways  of  doing  things. 
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•  People:  Not  all  people  are  innovative.  Those  that  are  must  be  identified,  supported, 
protected,  and  valued.  Bringing  together  the  right  mix  of  people  is  necessary  to 
thoroughly  explore  a  selected  idea,  and  this  team  needs  to  include  a  champion. 

•  Venue:  A  venue  is  not  a  specific  place  or  facility.  It  is  an  exploration  space,  which 
might  be  a  virtual  environment,  a  battlefield,  or  a  war  game. 

•  Experiments:  The  only  way  to  explore  the  complexities  of  a  system  is  through 
campaigns  of  experiments,  based  on  the  proper  venue,  people,  and  ideas.  Combining 
these  into  a  rigorous  program  of  technology  and  CONOPS  will  create  a  deep 
understanding  of  what  the  future  may  be  and  how  to  best  meet  it. 

Because  these  four  components  are  central  to  the  innovative  process,  we  next  explain 
them  in  further  detail. 


Ideas: 

Increasingly,  ideas  are  recognized  as  the  most  important  resource  of  any  organization 
whose  business  demands  staying  ahead  of  their  competition.  Innovative  organizations  actively 
seek  ideas  from  the  broadest  possible  base.  Such  organizations  instill  a  cultural  mindset  that 
actively  encourages  their  staff  to  challenge  the  basic  assumptions  on  which  the  currently 
accepted  ways  of  doing  things  are  based.  New  voices  in  the  organization  must  be  sought  out  and 
listened  to,  regardless  of  their  position  in  the  organization.  A  bottom-up  idea  generation  process 
is  essential  for  an  organization  to  innovate  effectively. 

This  requires  a  system  that  permits  all  levels  of  the  organization  to  actively  engage  in  the 
idea-generation  process,  to  participate  in  the  debate  over  their  merits,  and  to  explore  ways  of 
combining  the  best  aspects  of  widely  differing  ideas  into  potential  innovations.  Forced 
participation  does  not  work;  however,  active  encouragement  to  gain  broad  participation  is 
essential.  Approaches  that  simply  allow  ideas  to  be  submitted  are  not  sufficient.  Blogs  provide 
not  only  an  electronic  “suggestion  box”  but  also  serve  as  a  debate  space  in  which  these  ideas  can 
be  refined  and  allow  true  innovators  in  an  organization  to  be  identified.  Wikis  also  provide  a 
way  for  ideas  to  be  easily  transmitted.  This  is  one  area  where  the  adage  “if  you  build  it  they  will 
come”  is  true.  We  want  to  create  an  environment  that  the  most  creative  folks  will  thrive  in. 

Highly  networked  virtual  reality  games,  such  as  America’s  Army™,  Counter-Strike™, 
and  others,  are  approaching  a  level  of  realism  that  could  provide  an  entirely  new  way  of  broadly 
generating  and  testing  ideas  for  system-level  innovations.  Today,  America’s  Army™  - 
originally  developed  as  an  Army  recruitment  tool  -  has  approximately  8  million  users,  of  whom 
4.2  million  have  reached  the  competency  required  for  full  participation  in  over  2.1  million  games 
(“experiments”)  being  played  each  day  on  nearly  2,200  game  servers  spread  around  the  globe. 
Over  100,000  new  players  join  every  month;  they  are  predominantly  young,  creative,  and  bring 
with  them  perspectives  and  cultural  insights  to  which  access  is  otherwise  limited.  Counter- 
Strike™  is  even  more  impressive:  current  statistics  regularly  show  over  200,000  players 
simultaneously  playing  the  three  variants  of  Counter-Strike™  at  any  given  time,  accounting  for 
almost  70  percent  of  the  online  first-person  shooter  playing  audience,  and  amounting  to  over  4.5 
billion  minutes  of  playing  time  each  month,  making  it  the  most  popular  online  first-person 
shooter  game  in  history. 
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Today’s  games  could  be  expanded  to  give  players  the  unfettered  ability  to  define  their 
own  new  capabilities  and  CONOPS  -  limited  only  by  the  constraints  of  physics  -  and  to 
experiment  with  them  in  the  gaming  environment.  Software  tools  could  automatically  observe 
the  resulting  gameplay  to  discern  successful  patterns,  and  the  worldwide  network  of  discussion 
sites  on  which  players  collaborate  to  identify  successful  gaming  ideas  could  be  monitored  to 
track  evolving  concepts.  For  example,  it  is  imaginable  that  IEDs  would  have  appeared  in  such  a 
gaming  environment  long  before  they  did  in  the  “real”  world,  and  their  impact  could  have  been 
appreciated  on  the  basis  of  such  monitored  gameplay.  The  fidelity  of  reality  games  has  reached 
a  state  that  insights  from  such  experiments  are  credible,  and  if  even  a  tiny  percentage  of  the 
millions  of  such  “experiments”  being  played  every  day  produce  a  useful  insight,  the  effect  on  the 
pace  of  warfighting  innovation  could  be  enonnous. 

Regardless  of  where  they  come  from,  few  ideas  represent  viable  innovations  on  their  own 
in  the  form  in  which  they  are  proposed.  However  they  may  contain  a  key  concept  or  insight  that 
-  when  combined  with  other  ideas  -  can  lead  to  a  clearer  understanding  of  what  might  be 
possible,  or  provide  a  way  of  seeing  a  completely  different  approach  for  solving  a  problem. 
They  may  even  lead  to  the  solution  of  a  completely  different  problem.  Innovations  result  not  so 
much  from  the  original  ideas  themselves  but  from  putting  many  ideas  into  a  “mixing  bowl” 
environment  in  which  creative  teams  of  technologists  and  operators  can  combine  and  evolve 
them  to  discover  the  hidden  insights  they  may  hold.  A  similar  conclusion  was  reached  by  the 
2000  SAB  Battlelabs  Study  [7]. 

Consequently,  ideas  should  not  be  constrained  by  current  doctrines  or  requirements  of 
current  solution  approaches.  Nor  should  they  be  assessed  by  their  performance  in  relation  to 
metrics  established  for  completely  different  solution  approaches.  Most  new  ideas  will  perfonn 
worse  than  accepted  solution  paths  being  explored  from  the  current  status  quo.  The  idea  behind 
an  innovation  becomes  the  preferred  solution  approach  only  when  it  is  understood  in  tenns  of  the 
new  CONOPS  in  which  it  will  operate  and  in  the  context  of  new  metrics  appropriate  for  that 
CONOPS. 

People: 

Innovation  is  the  result  of  the  insightful,  collaborative  interactions  that  can  occur  when 
small  ad  hoc  teams  of  exceptional  innovators  are  brought  together  in  creative  venues.  These 
experimentation  teams  conduct  campaigns  of  system-level  discovery  experiments  that  explore 
novel  combinations  of  technology  and  CONOPS. 

Few  people  are  genuine  innovators,  and  only  genuine  innovators  should  be  assigned  to 
these  teams.  True  innovators  can  be  recognized  as  people  who  work  best  in  environments  where 
risk,  openness,  and  idea-sharing  are  the  norm;  where  ideas  outrank  seniority;  where  being  wrong 
is  not  a  failure;  where  learning  is  recognized  as  a  continual  process;  and  where  challenge  is 
viewed  as  the  highest  fonn  of  respect.  They  have  a  sense  of  urgency,  energy,  and  optimism. 
They  challenge  their  own  ideas  and  theories  as  much  as  those  of  others.  They  have  broad 
interests,  they  often  see  things  differently  than  the  rest  of  an  organization,  and  they  continually 
push  new  ideas  and  approaches  for  doing  things.  They  challenge  everything  -  even  processes 
that  appear  to  be  working  well.  As  a  result,  they  may  be  irritating  to  the  organization’s 
management,  are  often  silenced  by  the  mainstream  of  the  organization,  and  are  often  found  on 
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the  fringe  of  an  organization.  While  being  irritating  does  not  make  someone  an  innovator,  true 
innovators  are  likely  to  be  viewed  as  an  irritant  by  many  in  their  organization. 

A  process  for  discovering  and  cultivating  such  people  is  essential  for  an  innovative 
organization.  Those  with  true  innovator  personalities  suited  for  such  experimentation  teams 
must  be  actively  sought  out  and  recruited.  They  should  be  made  to  understand  that  being 
selected  to  serve  on  one  of  these  teams  is  a  reflection  of  their  abilities.  Service  on  these  teams 
must  become  seen  throughout  the  organization  as  a  prized  assignment,  and  as  part  of  building  the 
“culture  of  innovation”  it  must  be  acknowledged  as  such  by  the  highest  levels  of  the 
organization.  Those  who  prove  to  be  successful  as  members  of  these  ad  hoc  experimentation 
teams  should  be  tapped  again  when  appropriate  for  later  teams.  They  should  be  given  a 
continuing  sense  of  association  with  the  innovation  process.  They  must  be  tracked  through  their 
careers  to  protect  them  from  organizational  pressures  to  sideline  them,  and  their  innovation 
talents  should  be  actively  developed  over  the  course  of  their  career.  An  organization  that  fails  to 
cultivate  them,  or  that  fails  to  retain  such  people,  loses  what  may  be  its  most  valuable  asset. 

Experimentation  teams  must  consist  of  a  mix  of  technologists  and  operators.  System- 
level  experiments  require  working  at  the  interface  between  technology  and  CONOPS;  the 
solution  being  sought  will  rarely  be  a  purely  technological  one.  Bringing  technologists  and 
operators  together  early  in  the  process  allows  technologists  to  understand  the  operational  context 
of  the  problem  and  gives  operators  an  understanding  of  new  CONOPS  made  possible  by 
technology.  Team  integrity  must  be  maintained  throughout  the  experimentation  campaign. 
Bringing  new  members  onto  the  team  in  later  stages  of  the  process  risks  disrupting  the  deep, 
shared  understanding  of  the  problem  and  potential  system-level  solutions  that  have  been  gained 
by  team  members  over  the  course  of  the  experiment  campaign.  An  important  aspect  of  SLE  is 
finding  people,  both  technologists  and  operators,  who  are  capable  of  collaborating  about  a  future 
environment.  Many  people  have  a  vision,  but  many  are  unwilling  to  change  that  idea.  The 
notion  here  is  that  better  understanding  of  future  technologies  should  shape  future  operational 
concepts,  and  vice-versa. 

A  key  member  of  the  experimentation  team  is  the  “champion.”  Every  experimentation 
team  needs  a  high-level  champion  to  protect  the  experiment  campaign  from  pressures  to 
maintain  the  status  quo.  He  or  she  is  the  driving  force  behind  the  experiment  campaign,  sustains 
the  vision  for  innovation  throughout  the  campaign,  and  often  was  the  original  advocate  for  the 
concept  that  led  to  the  campaign.  While  the  initial  idea  or  concept  will  likely  evolve  -  perhaps 
dramatically  -  over  the  course  of  the  campaign,  the  champion’s  original  vision  of  a  system-level 
innovation  and  the  capabilities  it  could  offer  is  what  sustains  the  campaign.  Historically,  most 
innovations  have  had  an  identifiable  champion  behind  them.  The  champion  is  typically  the  first 
member  of  the  team  and  plays  a  key  role  in  selecting  and  guiding  the  rest  of  the  experimentation 
team.  It  is  important  to  recognize,  for  example,  that  in  a  commercial  organization  the  champion 
is  not  the  CEO,  but  it  is  someone  who  clearly  has  the  support  of  the  CEO. 

Another  key  member  of  the  experimentation  team  is  the  facilitator.  This  is  the  person 
who  understands  how  to  form  experiments  such  that  they  do  not  become  biased  by  how  we  do 
business  today.  This  is  a  very  tricky  process  to  figure  out  what  elements  of  how  we  do  business 
today  to  use,  and  which  ones  to  shed. 
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Venue: 

Innovation  teams  do  their  work  in  a  variety  of  venues;  these  are  not  necessarily  physical 
spaces,  but  rather  insight-promoting  spaces  where  effective  exploration  of  ideas  and  discovery  of 
hidden  insights  can  occur.  Venues  for  system-level  experimentation  are  different  from  the 
environments  used  for  training  exercises  or  for  demonstrations.  They  are  fundamentally  based 
on  challenge-competitive  gaming  environments  in  which  the  merits  of  ideas  for  the  interplay 
between  technology  and  CONOPS  can  be  explored.  The  venues  provide  the  “mixing  bowl” 
within  which  operators  and  technologists  interact  to  explore  and  evolve  ideas. 

The  most  productive  venue  must  be  used  for  each  stage  of  an  experiment  campaign,  and 
venues  will  differ  at  various  stages  of  the  campaign.  In  early  stages  of  experimentation,  simple 
“seminar  games”  allow  for  brainstorming  to  provide  an  understanding  of  broad  concepts  and 
allow  rapid  evolution  of  potential  solution  approaches.  Subsequent  stages  may  range  from 
simple  “maps  on  the  floor”  competitions  to  networked  reality  games  that  explore  the  effects  - 
both  intended  and  unintended  -  of  proposed  solutions  in  near-fully  operational  contexts.  The 
venues  for  final  stages  of  an  experiment  campaign  may  include  field  experimentation  or  even 
experimental  operational  implementation. 

Networked  reality  games  represent  a  rapidly  emerging  class  of  venues  that  could  become 
a  major  enabling  element  for  system-level  experimentation.  Experience  with  such  games  -  from 
DARPA’s  early  SIMNET  environment  to  today’s  massively  multiplayer  online  games 
(MMOGs)  -  shows  that  detailed  realism  in  the  venue  itself  is  not  needed  to  provide  high  fidelity 
in  such  games.  More  important  are  a  rapid  pace  of  operations,  the  almost  limitless  degrees  of 
freedom  and  resulting  fog-of-war  created  by  interactions  among  large  numbers  of  players,  and 
the  ability  to  work  in  a  highly  collaborative,  competition-based  environment.  Today  MMOGs, 
and  especially  team-based  tactical  first-person  shooter  games  such  as  Counter-Strike™,  provide 
venues  in  which  highly  complex,  collaborative,  and  competition-based  system-level  experiments 
could  be  conducted.  Adding  the  capability  in  such  venues  for  participants  -  both  friendlies  and 
adversaries  -  to  define  their  own  capabilities,  unfettered  by  doctrinal  or  cultural  limitations  and 
bounded  only  by  the  laws  of  physics,  would  enable  a  virtual  environment  for  true  system-level 
experimentation. 

An  important  consideration  with  respect  to  venue  is  to  make  sure  that  the  venue  does  not 
become  the  objective  of  the  team.  This  is  a  common  failure  in  many  organizations.  The  venue  is 
concrete  and  can  easily  become  the  focus  of  the  resources  and  activity.  SLE  promotes  the  notion 
that  the  venue,  while  important,  is  simply  a  means  to  an  end;  the  end  being  the  successful 
exploration  of  a  potential,  disruptive  innovation. 


Experiments: 

A  “campaign  of  experiments”  is  the  process  by  which  the  ad  hoc  experimentation  teams 
explore  ideas  for  the  interplay  between  technologies  and  CONOPS,  develop  the  insights  that 
produce  a  deep  understanding  of  potential  future  environments,  and  discover  combinations  that 
provide  innovative  solutions  to  a  problem.  For  any  given  problem,  the  campaign  is  a  sequence 
of  challenge-based  discovery  experiments  that  progresses  from  a  typically  simple  initial  venue  to 
a  final  one  in  which  the  proposed  solution  can  be  understood  in  a  near-fully  operational  context. 
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Frequent  experimentation  with  analysis  and  sharing  of  results  are  keys  to  achieving 
system-level  innovations.  Each  experiment  in  the  campaign  is  planned  based  on  the  experience 
and  insights  obtained  from  those  preceding  it.  Results  from  early  experiments  exploring  new 
solution  approaches  may  compare  poorly  when  measured  by  metrics  established  for  currently 
accepted  solution  paths.  Over  the  course  of  the  campaign,  the  succession  of  experiments 
explores  increasingly  deeper  aspects  of  the  problem  to  develop  a  clearer  understanding  of 
potential  solution  approaches.  A  role  of  the  “champion”  is  to  protect  the  experiment  campaign 
from  pressures  to  revert  to  the  status  quo  long  enough  for  the  SLE  process  to  work. 

Capturing  and  archiving  results  from  the  sequence  of  experiments  is  central  to  an 
effective  experimentation  campaign.  This  facilitates  analysis  and  insight,  which  allows  design  of 
the  next  step  in  the  campaign.  Data  mining  of  the  experiment  results  pennits  hidden  insights 
within  them  to  be  discerned.  For  experiments  involving  massively  multiplayer  online  game 
venues  such  as  Counter-Strike™,  where  200,000  or  more  players  worldwide  may  be  involved  in 
thousands  of  simultaneous  games,  the  amounts  of  observational  data  collected  by  monitoring 
gameplay  can  be  enormous.  Data  mining  can  then  have  potentially  profound  benefits;  the  range 
of  innovations  being  conceived  and  explored  in  such  games  can  far  exceed  what  an  ad  hoc 
experimentation  team  could  possibly  do. 


Contrasting  the  current  system  with  the  SLE  approach 

To  better  understand  the  SLE  process,  we  can  compare  it  to  the  Air  Force’s  current 
practices  and  procedures. 

The  Air  Force  currently  responds  to  defined  requirements  and  key  performance 
parameters  (KPPs),  limiting  its  ability  to  respond  quickly  to  new  challenges.  It  takes  years  to 
field  new  technologies  in  response  to  our  adversaries’  innovations.  By  creating  a  shared 
understanding  of  possible  future  environments  and  by  defining  technology  and  tactics  to  respond 
to  those  environments,  the  SLE  process  should  create  a  much  more  agile  and  responsive  Air 
Force. 

The  Air  Force  tends  to  commit  very  early  to  defined  concepts,  often  locking  in  place  an 
approach  that  is  not  optimal.  In  an  SLE,  the  concept  evolves  throughout  the  experimental 
campaign,  yielding  an  iterated,  and  generally  more  optimized,  result. 

As  in  most  hierarchical  organizations,  it  is  often  difficult  to  form  teams  from  different 
parts  of  the  Air  Force,  which  leads  to  “stove-piped”  thinking.  The  SLE  approach  is  focused  on 
selecting  teams  from  across  the  Air  Force,  emphasizing  combined  technologist/operator  teams 
that  generate  synergies  in  knowledge  and  experience. 

The  Air  Force  currently  performs  demonstrations,  not  experiments,  and  fails  to  capitalize 
on  lessons  learned.  SLE  incorporates  many  small  experiments  to  gain  understanding,  even  when 
the  experiments  do  not  turn  out  as  planned.  Air  Force  demonstrations  generally  incorporate  a 
constrained  and  predictable  adversary;  this  does  nothing  to  stimulate  innovative  responses.  SLEs 
embrace  the  idea  of  an  unfettered,  technically-skilled  adversary  to  “break  the  system.” 

Finally,  the  demonstrations  done  by  the  Air  Force  are  costly  and  inflexible,  with 
disproportionately  long  preparation  times  with  respect  to  execution  times.  This  programmatic 


Public  Release 
11 


Public  Release 


inertia  makes  it  difficult  for  researchers  to  respond  to  unanticipated  issues  or  challenges.  The 
SLE  process  is  inherently  flexible  with  short  approval  times  and  iterative  execution. 


Organization  for  innovation 

Building  on  the  best  industrial  practices,  we  recommend  that  the  Air  Force  create  the 
position  of  Chief  of  Innovation  (COI),  selected  by  and  reporting  directly  to  the  CSAF.  The  COI 
would  lead  a  small,  lightweight  organization  responsible  for  enhancing  innovation  in  the  Air 
Force.  The  COI  must  be  predisposed  to  disruptive  innovation,  serving  as  the  “Idea  Tsar”  and  a 
provocateur  of  disruptive  thought  and  a  harvester  of  creativity  of  the  Air  Force.  He  or  she  must 
have  authority  derived  from  a  direct  connection  to  the  CSAF.  The  COI  must  also  have  sufficient 
resources  and  the  flexibility  to  directly  fund  experiments.  Above  all,  the  COI  must  have 
patience;  innovation  is  not  done  on  demand. 

The  Office  of  Innovation  will  build  on  the  four  principles  of  innovation  suggested  in  this 
brief:  ideas,  people,  venue,  and  experimentation.  The  role  of  this  office  will  be  to  identify, 
mentor,  and  track  AF  innovators;  search  for,  select,  and  define  new  SFEs;  form  Innovation  Task 
Forces  (ITFs)  to  execute  SFEs;  start,  monitor,  and  stop  SFEs;  facilitate  experiments  and 
transition  the  results;  and  to  maintain  and  evolve  the  SEE  process.  Perhaps  most  importantly,  the 
Office  of  Innovation  will  be  a  magnet  for  innovators,  attracting  unsolicited  ideas  from 
throughout  the  Air  Force. 

The  ITFs  will  execute  experimental  campaigns.  Experience  dictates  that  a  champion  lead 
each  ITF  to  mentor  and  defend  the  team.  The  ITFs  define  and  conduct  the  SEE  campaign; 
observe,  capture,  and  analyze  the  results;  and  iterate  and  evolve  the  system-level  solution.  The 
COI  will  monitor  the  ITFs  and  enhance  or  disband  as  appropriate.  A  key  point  is  that  these  are 
to  be  real  experiments,  not  demonstrations.  Therefore,  some  experiments  will  not  work  as 
planned.  The  ITF  will  capture  the  lessons  learned. 


Potential  SLEs 

During  this  study  a  number  of  possible  SFEs  were  considered  that  could  be  executed  in 
the  immediate  future.  These  have  not  been  vetted;  it  is  beyond  the  scope  of  a  quick  look  study  to 
do  more  than  suggest  ideas  for  an  experiment.  The  team  had  neither  the  time  nor  resources  to 
plan  an  experiment  campaign,  and  the  team  lacked  the  authority  to  set  priorities  for  the  Air 
Force. 

1)  Find  Once,  Track  Forever  (Non-blinking  Eye  in  the  Sky):  Recent  technologies  (e.g., 
Angel  Fire)  offer  promise  of  continuous  tracking,  which  continues  to  be  a  challenge  for 
operators.  Consider  a  campaign  of  experiments  involving  Battlefield  Airmen  (BA)  and 
technologists  to  see  how  having  such  information  would  affect  the  CONOPS  for,  as  an  example, 
Air  Force  Special  Operations  Command  (AFSOC).  The  experiments  could  examine  the  kind  and 
quality  of  information  available  in  the  field,  the  interface  between  the  operator  and  the 
information,  and  etc.  The  result  would  be  the  development  of  new  CONOPS  as  well  as  a  better 
definition  of  the  necessary  technology. 
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2)  Cockpit  Selectable  Weapon  Effects :  How  would  the  ability  of  the  operator  to  select 
weapons  effects,  yield,  timing,  etc.,  affect  operations?  A  series  of  experiments,  again  involving 
operators  and  researchers,  could  define  such  a  capability’s  employment  benefits  and  the  enabling 
technologies. 

3)  Weight  Reduction  for  AFSOC  personnel  -  Battery  Elimination'.  AFSOC  personnel 
currently  carry  heavy  equipment,  increasing  their  combined  weight  to  well  over  300  pounds. 
Batteries  comprise  much  of  that  weight.  To  reduce  the  weight,  either  technological  solutions 
will  reduce  the  power  needs  (or  provide  lightweight  means  to  generate  power)  or  the  BAs  will 
change  their  operational  needs.  Experiments  that  assess  new  technologies  as  well  as  changes  in 
the  equipment  the  BAs  carry  could  help  define  both  new  ways  of  doing  their  business  as  well  as 
required  technologies. 

4)  Directed  Energy  (DE)  Technology  in  the  Joint  Urban  Fight :  The  impact  directed- 
energy  technology  will  have  in  an  urban  environment  is  not  yet  clear,  nor  is  it  clear  how  such 
applications  could  be  effectively  employed.  Experimentation  involving  researchers  and 
operators  are  critically  needed  to  work  out  new  tactics  and,  as  importantly,  help  delineate  the 
kinds  of  DE  technologies  that  could  be  most  effective. 

5)  Massively  Multiplayer  Online  Games :  America’s  Army™  today  has  over  7.2  million 
users;  with  4.2  million  having  the  competency  required  for  full  participation  in  over  2.1  million 
games  (“experiments”)  played  each  day  on  nearly  2,200  game  servers  spread  around  the  globe. 
The  Air  Force  should  create  a  similar  game  to  mine  the  world  for  new  ways  of  thinking. 

The  Air  Force  should  consider  all  of  these  possible  scenarios  as  SLEs  in  the  near  future. 
The  Air  Force  should  stage  a  competition  of  ideas  and  then  assemble  Innovation  Task  Forces  to 
execute  the  SLEs. 


Recommendations 

The  study  has  two  primary  recommendations: 

The  Air  Force  should  adopt  system-level  experimentation  as  a  process  by  which  it  can 
provoke  the  development  of  disruptive  innovations  to  meet  its  future  challenges.  The  team 
recommends  the  adoption  of  the  four  components  of  the  SLE  process  as  described  in  this  report 
and  the  encouragement  and  support  of  experimentation  that  combines  technologists  and 
operators.  By  exploring  technologies  and  CONOPS  for  possible  futures,  the  Air  Force  increases 
its  ability  to  predict  the  future  capabilities  of  adversaries  (and  to  develop  means  to  defeat  them) 
and  will  more  adeptly  create  new,  disruptive  capabilities  for  itself.  The  experimentation  must  be 
based  on  unfettered  competition,  to  simulate  the  “fog  and  friction  of  war.”  The  team  urges 
pervasive  adoption  of  the  SLE  process  throughout  the  Air  Force. 

To  facilitate  and  enhance  the  SLE  process,  the  Air  Force  should  create  a  Chief  of 
Innovation  position.  This  person,  whether  military  or  civilian,  should  be  selected  by  and  report 
to  the  CSAF.  The  COI  must  be  someone  predisposed  to  innovation.  The  CSAF  should  consider 
someone  from  industry  with  a  proven  track  record.  The  COI  must  have  authority  to  carry  out 
his/her  mission  as  well  as  the  necessary  resources  to  support  SLEs.  The  COI  will  be  responsible 
for  coordinating  with  existing  organizations  to  identify  SLE  opportunities  and  needs,  organizing 
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implementation  of  selected  SLEs,  evaluating  SLE  results,  assessing  and  facilitating  transition 
opportunities,  and  maintaining  the  knowledge  database. 

The  SLE  team  suggests  that  one  way  to  implement  these  recommendations  is  to  designate 
the  AU/CC  as  COI  with  execution  coordinated  by  the  Center  for  Strategy  and  Technology 
(CSAT)  and  with  active  involvement  of  AFRL.  This  is  consistent  with  practices  described  in  the 
report  (e.g.,  report  to  CEO,  disruptive  innovation  explored  outside  core  processes).  It  is  also 
consistent  with  the  recent  decision  to  form  an  Air  Force  Futures  Group  between  AF/A8  and  AU. 


Summary 

To  recreate  its  ability  to  lead  the  development  and  application  of  disruptive  innovation 
the  Air  Force  must  foster  a  culture  that  promotes  innovation.  Such  a  culture  thrives  on  ideas  that 
challenge  conventional  thought  and  doctrine  harvested  from  all  sources  in  and  out  of  the  Air 
Force.  It  is  essential  to  find,  recruit,  develop,  and  protect  true  innovators  from  throughout  the 
organization.  Most  importantly,  however,  the  Air  Force  needs  to  embrace  experimentation  as  a 
principle  means  of  provoking  disruptive  innovation.  They  need  to  conduct  campaigns  of 
discovery  experiments  outside  of  the  core  process  that  involve  both  technologists  and  operators 
to  co-evolve  technology  and  CONOPS.  They  should  include  unconstrained  adversaries  to  mimic 
the  “friction  of  war”  and  to  provoke  creative  tactics.  They  need  patience  and  to  recognize  that  in 
an  experiment,  failure  only  occurs  when  we  do  not  leam  something.  Finally,  leadership  in 
innovation  must  start  with  the  CSAF,  who  should  appoint  a  Chief  of  Innovation  to  serve  as  the 
“Idea  Tsar”  and  principle  innovator. 
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Annotated  Brief 


Air  Force  Scientific  Advisory  Board 


System-Level  Experimentation 

discover ,  explore,  and  understand  game-changing  ways 
...  to  fly  and  fight  in  Air.  Space,  and  Cyberspace 

SAB  Summer  Session 
30  June  2006 

Co-chairs 

Steve  Cross  and  Scott  Fouse 


There  are  five  key  points  in  this  study: 

1 .  System-level  experimentation  (SLE)  is  an  approach  to  unfettered  exploration  of 
alternative  system  concepts  focused  in  future  environments  to  aid  in  earlier  discovery 
of  game-changing  ways  to  “fly  and  fight.” 

2.  Our  adversaries  innovate  while  we  merely  react  -  a  more  proactive  approach  would 
support  system-level  discovery  experiments  focused  on  disruptive  ideas,  thus 
increasing  the  likelihood  of  finding  game-changing  ways  to  “fly  and  fight”  BEFORE 
the  fight  thereby  forcing  our  adversaries  to  react  to  our  innovations. 

3.  At  one  time  the  USAF  aggressively  searched  for  “game-changers,”  arguably  it  has 
not  for  many  years. 

4.  The  USAF  is  not  alone;  this  is  also  an  important  national  issue  for  U.S.  industry  -  but 
there  has  been  impressive  progress  in  the  last  year,  and  there  are  lessons  learned  the 
USAF  can  apply  now. 

5.  Ad  hoc  groups  (e.g.,  “innovation  task  forces”)  drawn  from  Air  Force  organizations 
can  be  formed  to  do  this  -  such  needs  to  be  supported  by  a  small  group  of  facilitators; 
they  need  to  be  resourced,  protected,  and  visibly  important  to  the  Chief  of  Staff. 
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National  safety  would  be  endangered  by  an  air  force 
whose  doctrines  and  techniques  are  tied  solely  to  the 
equipment  and  processes  of  the  moment. 


...  any  air  force  which  does  not  keep  its  doctrines 
ahead  of  its  equipment,  and  its  vision  far  into  the 
future,  can  only  delude  the  nation  into  a  false  sense 

of  security. 

Gen  Henry  “Hap”  Arnold 
November  1945 


This  study  is  about  provoking  disruptive  innovations  through 
system-level  experimentation 


Even  sixty-one  years  ago  leaders  recognized  the  importance  of  understanding  technology 
and  tactics  required  to  dominate  in  future  environments.  In  today’s  parlance,  General  Arnold  is 
talking  about  maintaining  the  USAF’s  combat  edge  through  innovation  (e.g.,  pursuing 
revolutionary,  game-changing  systems).  In  the  commercial  world,  innovation  means  staying 
competitive;  in  the  military  world  it  means  being  dominant.  This  is  something  the  early  Air 
Force  leadership  was  concerned  about  just  as  the  CSAF  is  today. 

Air  Marshall  Mason  makes  some  interesting  observations  about  this  famous  quote  in 
“Innovation  and  the  Military  Mind”  (Mason  1986).  He  cautions  that  not  focusing  on  innovation 
means  becoming  predictable  and  therefore  vulnerable. 

Even  the  most  cursor) >  survey  of  military >  history >  illustrates  the  critical  importance  of  technological 
and  tactical  innovation.  The  stirrup,  the  longbow,  barbed  wire,  the  tank,  blitzkrieg,  radar, 
electronic  countermeasures,  AWACS,  helicopter  assault,  and  the  astonishing  aggregate  of  British 
innovation  displayed  during  the  Falklands  War  are  random  examples.  Sometimes  the  vision  of  the 
innovators  has  outrun  the  capability  of  technology >:  the  early  submariners,  the  early  aircraft  carrier 
advocates,  the  first  air  power  theorists,  the  proponents  of  surface-to-air  missiles,  and,  just  possibly, 
those  enthusiasts  who  unreservedly  espouse  the  cause  of  enhanced  technology >  as  the  panacea  for 
today's  Western  strategic  dilemmas  might  be  so  categorized.  Yet  without  such  visionaries  and 
without  innovation,  a  nation 's  way  of  war  becomes  predictable;  and  predictable  means  vulnerable. 

While  most  USAF  experimentation  today  deals  with  demonstrations  (e.g.,  JEFX,  ATDs), 
the  focus  of  this  study  is  on  discovering,  exploring,  and  understanding  “game-changing”  ways  to 
“fly  and  fight  in  air,  space,  and  cyberspace.”  What  is  suggested  in  this  study  is  not  new  given  the 
history  of  the  Air  Force;  in  fact,  it  is  strikingly  similar  to  how  experimentation  was  pursued  in 
the  Air  Corps  Tactical  School  in  the  1930s  and  in  other  case  studies  presented  herein.  The  U.S. 
Air  Force  has  gotten  away  from  this  kind  of  experimentation.  It  is  time  to  reclaim  our  heritage! 
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Our  AF  Heritage  is  Innovation 


Common  characteristics 


Led  by  innovators  -  risk  taking  champions  with  a  vision 

Provoked  a  renaissance  of  ideas 

Created  and  used  a  mixing  bowl  (e. g.,  ops-tech  teams) 

Conducted  discovery  experiments  focused  on  future 
needs 

Enabled  unfettered  exploration  in  the  right  venue 

Rarely  did  the  formal  Requirements  Process  help;  more 
often  it  impeded  progress 


Early  experimentation  played 
a  key  role 


Innovation  is  not  new  to  the  Air  Force;  on  the  contrary,  innovation  is  its  heritage  (i.e.,  a 
fundamental  component  of  the  “Air  Force  gene  pool;”  if  you  will  the  Air  Force’s  “organizational 
DNA”).  This  study  highlights  the  innovators  listed  on  the  chart  because  they  had  a  vision  and 
they  used  experimental  approaches  to  find  ways  to  achieve  that  vision.  Regardless  of 
programmatic  and  cultural  obstacles,  to  say  nothing  of  the  personal  risks  to  their  careers,  they 
were  determined  to  explore  the  operational  benefits  their  tactical  and  technological  ideas  could 
bring  to  the  Air  Force.  To  test  their  theories,  they  pursued  discovery  experiments  to  understand 
the  future  and  to  develop  system  concepts  that  enabled  the  Air  Force  to  dominate  in  that  future. 
Case  studies  are  presented  in  Appendix  E  of  this  report.  An  excellent  description  of  innovation 
involving  experimentation  from  the  early  days  of  the  Air  Corps  is  in  Perry’s  1999  paper. 
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Outline 


a  Who  we  are  and  who  we  talked  to 

■  System-level  experimentation  (SLE) 

■  Definition 

■  Provoking  ‘game  changing’  innovation 

■  Approach 

■  Example  SLEs 

■  Summary  of  key  points 

■  Recommendations 


The  briefing  was  designed  to  cover  the  objectives  in  the  Terms  of  Reference  (ToR)  of 
this  study.  These  objectives  are  listed  below. 

•  An  overview  of  current  Air  Force  “state  of  practice”  and  published  approaches  for 
exploratory  experiments  involving  technologists  and  operators. 

•  An  overview  of  approaches  practiced  by  others  (e.g.,  other  services,  national 
laboratories,  commercial  industry,  and  adversaries). 

•  A  description  of  an  approach  by  which  system-level  experiments  could  be  used  to 
“stress  test”  technology,  focus  high  payoff/high  risk  research,  as  well  as  envision  and 
understand  new  operational  and  system  concepts. 

•  A  description  of  three  system-level  experiments  in  the  domain  of  Air  Force  special 
operations. 

•  Recommendations  for  a  potential  follow-on  study  to  address  the  scale-up  of  the 
approach  and  enhancements  to  technology  transition  processes  to  fully  realize  the 
benefits. 

Note:  The  last  objective  was  not  addressed  explicitly  in  this  study.  However,  at  the  time 
of  the  study’s  completion,  there  were  several  SAB  studies  proposed  for  consideration  in  FY07 
that  would  include  an  SLE  approach.  This  study  builds  on  several  recent  SAB  studies: 

•  USAF  Battlelabs  (2000) 

•  Air  Force  Operations  in  Urban  Environments  (2005) 

•  System-of-Systems  Engineering  for  Air  Force  Capability  Development  (2005) 
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Study  Members 


Air  Force  SAB 
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■  Dr.  Stephen  Cross 
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■  Dr.  Ken  Ford 

■  Mr.  Scott  Fouse 

■  Dr.  Richard  LeSar 


Stanford  University  (Physics) 

Georgia  Institute  of  Technology  (IT,  software  engineering) 
University  of  Michigan  (Propulsion) 

Institute  for  Human  and  Machine  Cognition  (Al,  HCI) 

ISX  Corporation  (Systems,  IT) 

Los  Alamos  National  Laboratory  (Materials) 


Maj  Gen  Eric  Nelson,  USAF  (Ret)  Independent  Consultant  (Systems,  Software) 
Dr.  Brad  Parkinson  Stanford  University  (GPS,  Sensors,  Systems) 

Mr.  Skip  Saunders  Independent  Consultant  (Systems,  IT) 


■  Executive  Officers  and  SAB  Office  Support 


■  Major  Valerie  Manning 

■  Major  Lee  Chase 

■  Captain  Justin  Joffrion 

■  Major  Matt  Keihl 

■  Mr.  Justin  Waters 


SAB  (Aerospace  Systems) 
AFRL/PR  (Propulsion) 

USAF  Academy  (Economics) 
AFRL/DE  (Directed  Energy) 

SAB  (Analyst  and  Technical  Writer) 


a  special  thanks  to  Maj  Gen  George  B.  Harrison,  USAF  (Ret) 


The  study  members,  including  the  executive  officers  and  professional  SAB  staff,  were 
diverse  in  terms  of  technical  and  operational  experience.  This  diversity  directly  led  to  the  ideas 
presented  in  this  study.  A  key  lesson  from  industry  best  practice  with  respect  to  innovation  is  to 
engage  groups  with  diverse  experiences.  If  one  asks  a  homogeneous  group  a  question,  one  gets 
the  same  answer.  If  one  asks  a  group  with  diverse  experiences  the  same  question,  one  gets  a 
variety  of  answers.  In  that  variety  will  be  likely  “out  of  the  box”  ideas  that  lead  to  innovation. 
The  study  team  was  purposely  designed  to  leverage  this  key  lesson. 

The  study  members  appreciated  the  opportunity  to  conduct  a  “quick  look”  study  and 
make  recommendations  to  the  Air  Force  on  a  very  important  topic  at  a  very  critical  time.  Each 
contributed  greatly  to  the  study  results. 
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Contact  List  | 

AIR  FORCE 

SecAF 

AIR  FORCE  SCHOOLS 

HQ  USAF/A2/A8XC  (Blue  Horizons) 

AU/CC 

SAF/AQR 

AWC/CC 

SAF/XCXX/XCOI 

-  Center  for  Strategy  &  Technology 

AMC/A58P 

Air  Command  &  Staff  College  (ACSC) 

AFSOC/CC  -  A2-A6 

School  for  Advanced  Airpower  Studies 

SMC/CV 

USAFA 

AFRL/CC/CT 

OTHER  GOV’T 

-  Directorates:  VA,  PR,  SN,  HE,  VS,  DE,  IF,  MN 

USMC 

AFMC/A8 

NWDC/CC,  Chief  Engineer 

AFC2ISRC 

ONR,  Chief  Scientist 

Battlelabs  ■  C2,  UAV,  Air  Warfare 

MilSATCOM  JPO 

Rapid  Capabilities  Office  (RCO) 

JFCOM/J9  and  JFIC 

INDUSTRY 

TRADOC 

Institute  for  Human  &  Machine  Cognition 

Los  Alamos  National  Laboratory 

Cisco 

Sandia  National  Laboratories 

IBM,  VP  Innovation  &  Technology 

NASA 

Council  on  Competitiveness 

Through  a  series  of  interviews,  round  table  discussions,  and  competitive  brainstorming 
sessions*  occurring  between  December  2005  and  June  2006,  members  of  the  study  panel  talked 
with  representatives  from  each  of  the  organizations  listed  above.  This  represented  another  key 
lesson  in  innovation  best  practice  -  “cast  the  net  wide”  throughout  as  wide  a  community  as 
possible. 

^Competitive  brainstonning  sessions  were  held  during  many  of  the  meetings.  Air  Force 
attendees  and  SAB  study  teams  were  subdivided  into  subgroups  and  given  the  task  to  design  an 
experiment  related  to  the  topic  under  discussion  (e.g.,  high  payoff  applications  when  the  team 
visited  AFSOC  and  high  payoff  directed  energy  applications  when  the  team  visited  the  Directed 
Energy  Director  of  AFRL).  Groups  presented  their  experiments  in  a  contest  where  the  best  was 
chosen  by  a  third  party  observer.  Competition  was  fierce.  Such  competitions  provoke  good 
ideas;  ideas  that  might  not  otherwise  be  envisioned  through  a  more  ‘group  think’  venue.  Such 
sessions  were  themselves  another  test  of  another  best  practice  idea  for  innovation  -  create  venues 
where  it  is  permissible  to  think  risky  thoughts  and  to  creatively  explore  ideas  without  fear  of 
failure. 
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System-Level  Experimentation  (SLE) 


Campaigns  of  experiments 
to  explore  system 
alternatives  through 
innovative  combinations 
of  technologies, 
CONOPS ,  and 
organizations  in  a  future 
environment 


Technology 


Experimentation 


Increases  ability  to  discover  game-changing  ways  to  “fly  and 
fight”  BEFORE  the  fight 


In  this  study,  “system”  is  a  combination  of  technology,  processes  (e.g.,  CONOPS), 
people,  and  an  organizational  construct  to  perform  some  set  of  functions.  This  short  hand 
definition  follows  from  one  offered  by  INCOSE: 

A  system  is  a  construct  or  collection  of  different  elements  that  together  produce  results  not  obtainable 
by  the  elements  alone.  The  elements,  or  parts,  can  include  people,  hardware,  software,  facilities, 
policies,  and  documents;  that  is,  all  things  required  to  produce  systems-level  results.  The  results 
include  system-level  qualities,  properties,  characteristics,  functions,  behavior,  and  performance.  The 
value  added  by  the  system  as  a  whole,  beyond  that  contributed  independently  by  the  parts,  is 
primarily  created  by  the  relationship  among  the  parts;  that  is,  how  they  are  interconnected. 

SLE  involves  a  campaign  of  discovery  experiments  (i.e.,  a  series  of  experiments  intended 
to  explore  and  understand)  that  combine  technology  (both  developed  and  envisioned)  and 
CONOPS  to  develop  new  capabilities.  SLEs  can  be  used  to  explore  a  wide  range  of  possible 
future  environments  and  system  alternatives  in  those  environments.  By  exploring  technologies 
and  CONOPS  for  possible  futures,  the  Air  Force  increases  its  ability  to  project  the  future 
capabilities  of  adversaries  and  to  develop  means  to  defeat  them  as  well  as  being  able  to  create 
new,  disruptive  capabilities  for  itself.  SLEs  are  one  type  of  experiment  that  compliment  other 
forms  used  in  the  Air  Force  today. 

Key  words  and  phrases  unique  to  SLE  and  discussed  in  more  detail  in  the  report  include: 

•  Provoking  ideas  through  competition;  unfettered  exploration  of  ideas 

•  An  unconstrained  adversary  (e.g.,  red  team) 

•  A  series  or  campaign  of  discovery  experiments 

•  Success  includes,  in  fact  starts  with,  understanding  failure 

•  The  outcome  is  a  deep  understanding  of  a  future  environment 
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SLE  Is  Not ... 


■  JEFX  -  large  bi-annual  demonstration  experiment 

m  SLE  involves  frequent,  smaller  scale  discovery  experiments 

■  Battlelab  -  near-term  focus  (“Kenney  initiatives”) 

m  SLE  is  focused  on  system  experiments  in  “Mitchell  initiatives” 

■  Big  Safari,  RCO*  -  rapid  fielding  for  high  priority  needs 

■  SLE  is  focused  on  understanding  the  future  -  unknown  needs 

■  A8’s  Future  Game  -  analysis  of  future  environments 

■  SLE  is  a  campaign  of  system  experiments  in  future  environments 

■  AFRL  -  technology  development  to  meet  Air  Force  needs 

■  SLE  could  be  supported  by  AFRL  to  explore  future  systems 

■  Air  University  -  education/training  to  meet  Air  Force  needs 

■  SLE  could  be  supported  by  A  U  to  explore  future  systems 

*Rapid  Capabilities  Office 


SLEs  can  be  compared  and  contrasted  with  ongoing  forms  of  experimentation  and  other 
organizational  pursuits.  The  focus  is  discovery  (as  opposed  to  demonstration)  in  the  context  of 
what  were  originally  called  “Mitchell  initiatives”  (i.e.,  revolutionary)  when  the  Battlelabs  were 
first  formed.  The  products  of  SLEs  may  or  may  not  be  a  prototype  or  working  system,  but  rather 
a  better  understanding  of  a  future  environment  (e.g.,  the  discovery  of  unknown  needs).  As  will 
be  discussed,  SLEs  could  be  conducted  within  wargames,  but  they  are  not  a  wargame  per  se. 

Alberts  and  Hayes  (2005),  in  their  book  about  campaign  of  experiments,  define  three 
types  of  experiments: 

•  Discovery  experiments  -  designed  to  generate  new  ideas  or  ways  of  doing  things 

•  Hypothesis  testing  experiments  -  the  classic  type  used  by  scholars  and  researchers  to 
advance  knowledge  (by  seeking  to  falsify  specific  if  /  then  statements)  or  to  discover 
their  limiting  conditions 

•  Demonstration  experiments  -  create  a  venue  in  which  known  truth  is  recreated;  most 
Air  Force  experiments  fall  into  this  category 

•  In  addition,  they  define  a  campaign  of  experiments  as  a  “set  of  related  activities  that 
explore  and  mature  knowledge  about  a  concept  of  interest.” 
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Innovation  Types  CO 

Sustaining 

Disruptive 

Innovation 

Innovation 

“Play  the  Game  Better” 

“Change  the  Game” 

Requirements  focused 

Experimentally  derived 

Examples 

Examples 

Latest  PC 

First  PC 

Latest  Cell  Phone 

iPOD 

Latest  Automobile 

Wikis 

BAO*  Toolkit 

Southwest  Airline’s  Biz  Model 

*Battlefield  Airman  Operations 

There  are  basically  two  types  of  innovation:  sustaining  and  disruptive.  Clayton 
Christensen,  author  of  The  Innovator's  Dilemma,  The  Innovator’s  Solution,  and  many  articles  on 
innovation  in  recent  years,  used  the  concept  of  a  disruptive  technology  to  describe  a  new 
technological  innovation,  product,  or  service  that  eventually  overturns  the  existing,  dominant 
technology  or  product  in  the  market.  He  broadened  that  concept  to  disruptive  innovation  when 
he  recognized  that  few  technologies  are  intrinsically  disruptive;  rather,  their  use  creates 
disruptive  impacts. 

Sustaining  innovation  is  the  successive,  incremental  improvement  to  the  performance  of 
existing  products.  Examples  of  sustaining  innovation  are  common  -  several  examples  are  listed. 
Most  Air  Force  technologies  currently  in  development  are  sustaining  in  nature. 

In  contrast,  disruptive  innovation  can  originate  from  new  technologies  or  processes  to 
produce  a  radical  new  capability.  Several  examples  are  listed.  Disruptive  innovation  is  the 
formal  name  in  the  literature  for  “game-changer.” 

Key  insight:  Game-changers  are  not  the  result  of  invention  (e.g.,  a  technological 
advance),  but  the  use  of  that  invention.  Co-evolution  and  experimental  assessment  of  system 
technology  and  its  possible  use  in  future  environments  are  how  game-changers  are  discovered. 
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Disruptive  innovation  in  Warfare 


Throughout  history,  risk  taking  champions  with  a  vision  provoked  a 
renaissance  of  ideas  that  led  to  game  changing  ways  to  dominate 

Early  Warning  System  -  Chennault  (1933) 

Carrier  Warfare  -  US  Navy  (1930s) 

Blitzkrieg  -  Guderien  (1940s) 

ICBM  -Schriever  (1950s) 

Space-based  Surveillance  -  (1960s) 

Precision  Guided  Weapons  -  (1970s) 

Stealth  -(1970s) 

Armed  Predator  -  (2002) 

None  of  these  resulted  from  the  formal  Requirements  Process 


Technology  alone  does  not  produce  innovation 


Military  history  contains  many  examples  of  disruptive  innovation.  Each  of  these 
disruptive  innovations  relied  partially  on  technology  and  partially  on  the  creative  insights  shared 
between  operators  and  technologists  on  how  best  to  use  that  technology.  The  military  did  not 
develop  these  solutions  as  products  of  a  formal  acquisition  process;  rather,  innovators 
incorporated  technologies  and  ideas  then  experimented  with  tactics  and  processes  to  change  the 
conduct  of  war. 

A  recent  example  involves  the  armed  UAV.  The  following  quote  from  a  speech  by  Air 
Force  Chief  of  Staff  General  T.  Michael  Moseley  to  the  American  Enterprise  Institute  on  1 1 
October  2005  should  help  motivate  the  need  for  SLE: 

Having  been  the  wing  commander  that  got  the  Predator  first  and  having  had  the  Chief  of  Staff  at 
the  time,  General  Fogleman,  say,  “You’ve  got  these  things,  ”  and  I  said  “Boss,  what  do  you  want 
me  to  do  with  them?’’  He  said,  “Go  fly  them  and  figure  out  how  they  work.  ’’  I  said  “I'm  a  little 
busy  out  here.  If  this  is  a  hobby  project  let  me  know  and  I'll  give  it  some  good  thought,  but  you 
really  want  me  to  take  these  things  and  figure  out  how  to  fly  them?’’  He  said,  “Yeah,  just  go 
figure  it  out.  ’’  I  became  a  big  fan,  and  I  became  a  big  fan  of  these  things  in  combat,  and  I 
became  a  big  fan  especially  when  you  can  hook  infrared  imaging  to  a  Hellfire  missile  or  to  a 
laser-guided  500-pounder  off  this  thing.  So  not  only  can  you  go  out  and  look  for  something,  you 
can  ID  it  and  you  can  whack  it. 

This  innovation  described  by  General  Moseley  will  be  used  later  to  illustrate  how  SLEs 
can  facilitate  disruptive  innovation. 
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AF  Innovation: 

What  We  Saw  and  Heard 

■  Mostly  demonstration  experiments  tied  to  requirements 

■  Institutionalized  expertise  in  sustaining  innovation 

■  Slow  response  to  the  enemy’s  rapid  innovations 

■  Few  discovery  experiments  focused  on  game  changing 

■  Many  Airmen  predisposed  to  both  types  of  innovation 

■  Culture  and  policies  discourage  disruptive  innovation 


Air  Force  must  again  become  a  leader 
in  disruptive  innovation. ..or  we  risk  becoming  irrelevant 


The  Air  Force  is  innovative,  but  its  institutionalized  expertise  lies  in  sustaining 
innovation  -  a  capability  it  should  not  lose.  Well-known  issues  with  the  acquisition  process 
often  make  the  timely  deployment  of  advances  a  challenge.  Even  when  faced  with  an  innovative 
adversary  (e.g.,  improvised  explosive  devices),  the  cumbersome  processes  inhibit  a  quick, 
effective,  and  decisive  response.  Big  Safari  and  RCO  are  current  day  “skunk  work”  approaches 
that  accelerate  the  development  and  fielding  of  solutions  to  high  priority  needs.  These  solutions 
tend  to  be  sustaining  innovations. 

The  Air  Force  has  struggled  to  develop  or  recognize  disruptive  innovation,  partially  due 
to  a  lack  of  focus  on  true  discovery  experiments  and  partially  due  to  exercises  being  conducted  to 
demonstrate  and  train.  Red  Force  ROE  are  often  constrained  to  achieve  training  and 
demonstration  objectives.  Allowing  unfettered  exploration,  including  unimpeded  involvement  of 
a  red  team  (similar  to  a  Red  Flag  Aggressor  Squadron),  is  not  typically  allowed.  Limiting  the 
full  exploration  of  a  solution  space,  including  what  an  intelligent  adversary  could  do  to  inhibit 
operations,  would  dilute  any  attempt  to  develop  a  deep  understanding  of  a  future  environment. 

However,  the  Air  Force  has  people  in  every  organization  the  study  team  visited  that  are 
predisposed  to  both  kinds  of  innovation.  Those  predisposed  to  disruptive  innovation  were 
constrained  in  that  pursuit  by  culture  and  policy  that  inhibited  the  free  exchange  of  ideas, 
displaying  intolerance  for  risk,  and  the  accompanying  negative  career  and  organizational  budget 
impacts  of  real  or  perceived  failure. 

The  Air  Force  is  capable  of  disruptive  innovation,  but  it  has  largely  lost  its  ability  to 
foster  it.  Expectations  about  future  warfighting,  however,  strongly  suggest  that  the  Air  Force 
must  again  be  a  leader  in  disruptive  innovation.  Some  solace  may  be  found  in  the  fact  that  many 
large  organizations  in  America  today  are  facing  a  similar  challenge. 
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Industry  Renaissance  in  Innovation 


National  Innovation  Initiative,  Dec  2004 


Industry  Best  Practices 


Business  Week,  Jun  2006 


•  Search  for  ideas  everywhere 

■  Assess  ideas  outside  core 
processes 

■  Identify  and  protect  people  who  are 
willing  to  challenge  conventional 
wisdom 

■  Leadership  in  innovation  must  start 
with  the  CEO 


“Making  innovation  work  is  the  single  most  important 
business  challenge  in  our  era” 


A  national  concern  expressed  two  years  ago  and  highlighted  in  the  President’s  2006  State 
of  the  Union  message  dealt  with  the  ability  of  U.S.  industry  to  innovate.  Several  recent  studies 
(see  references)  have  been  conducted  on  this  topic,  and  within  the  time  frame  of  this  SAB  study 
best  practices  have  begun  to  be  published. 

Four  key  best  practices  for  encouraging  innovation  within  an  organization  are  listed  on 
the  chart.  The  role  of  the  CEO  cannot  be  overstated.  The  CEO  grants  pennission  for  trying  “out 
of  the  box”  ideas,  for  learning  from  failure,  and  for  thinking  risky  thoughts.  Another  insight, 
also  from  Christianson  in  a  recently  published  handbook  on  innovation,  is  a  practice  followed  by 
many  CEOs  who  create  a  small  organization  outside  core  processes  with  the  task  to  pursue 
disruptive  ideas.  Christianson  recommends  that  organizations  do  this  when  their  culture  and 
processes  hinder  disruptive  thought. 
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The  graphic  is  adapted  from  Clayton  Christensen’s,  The  Innovator’s  Dilemma.  The 
graphic  illustrates  the  potential  impact  of  technology-based  disruptive  innovation  by  notionally 
plotting  the  performance  of  a  product  or  a  system  (y  axis)  versus  time  (x  axis).  The  key  idea  is 
that  sustaining  innovation  facilitates  a  steady  increase  in  product  performance  over  time  in 
pursuit  of  known  requirements.  Successful  incumbents  within  the  market  place  rarely  recognize 
the  next  best  thing.  They  are  focused  on  incremental  improvement  to  satisfy  market  needs  (or  in 
the  military  parlance,  key  perfonnance  parameters). 

Disruptive  innovation  occurs  when  someone  comes  along  with  a  new  idea.  Often  the 
initial  performance  is  below  marketplace  expectation.  The  innovation  is  pursued  “under  the 
radar”  until  it  matures  enough  to  leap  over  the  existing  system’s  capabilities  and  begin  its  own 
sustaining  innovation  path. 

The  Armed  Predator  mentioned  a  couple  of  charts  back  is  a  good  example.  The  Predator 
was  gaining  an  established  position  in  the  Air  Force  portfolio  as  a  surveillance  platfonn  when 
General  John  P.  Jumper,  then  Air  Force  Chief  of  Staff,  tasked  Lieutenant  General  Stephen 
Plummer,  who  was  Principal  Deputy  to  the  Assistant  Secretary  of  the  Air  Force  for  Acquisition, 
to  talk  to  officials  at  Wright-Patterson  AFB  about  adding  a  hellfire  missile  to  the  UAV.  To 
introduce  stress  into  the  system,  General  Jumper  gave  the  developers  90  days  and  $3M.  Fie 
believed  that  given  the  proper  motivation,  adequate  resources,  permission  to  fail,  and  their 
innovative  culture,  they  could  accomplish  the  task  -  and  they  did!  They  gave  the  Air  Force  a 
whole  new  capability. 

The  SLE  approach  is  to  integrate  a  series  of  discovery  experiments  with  commercial  best 
practices  for  innovation.  It  is  suggested  that  by  doing  this  the  Air  Force  can  institutionalize  a 
capability  to  support  and  exploit  disruptive  innovation. 
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Elements  of  SLE 


■  Ideas  -  cast  the  net  wide;  challenge  all  existing  doctrinal  limits 

■  People  -  exceptional  innovators;  driven  to  challenge  convention 

■  Venue  -  a  challenge-competitive  exploration  space 

■  Experiments  -  iterative  campaigns  of  discovery  experiments 


Ideas 


“Campaign  of  Discovery  Experiments” 


Ideas 

Ideas  ldeas 

Ideas 

ldeas  Ideas 

Ideas 

Ideas  Ideas 

Ideas 


Disruptive 

Innovation 


Ideas 


Ideas 


People 


Venues  Experiments 


Creates  deep  understanding  of  future  environments 


Four  essential  components  in  the  development  of  disruptive  innovation  through  system- 
level  experimentation  are  listed  on  this  chart.  Each  component  is  discussed  in  more  detail  in  the 
following  charts.  The  key  output  of  these  experiments  is  not  necessarily  a  prototype  or  a 
hardware  solution  but  a  deeper  understanding  of  the  future  environment. 
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Ideas 


■  Cast  the  net  wide;  challenge  all  existing  doctrinal  limits 

■  Don’t  constrain  by  known  requirements 

■  Listen  to  new  voices  -  leave  ego 
and  rank  at  the  door 

■  Promote  innovation  as  a  cultural 
mindset,  driven  by  insight 

■  Challenge  ideas,  results,  and 
theories  -  yours  and  others 

Use  online  simulations  and  internet 
software  tools  (e.g.  wikis,  blogs) 


Seek  diversity  of  ideas  and  encourage  open  debate 


Cybercraft 


Ideas  are  the  JP8  of  innovation.  Ideas  by  themselves  do  not  produce  innovation.  They 
must  be  captured,  debated,  and  applied  in  an  unconstrained,  collaborative  environment  to  have 
the  desired  disruptive  impact.  Ideas  should  be  solicited  from  as  wide  a  community  as  possible, 
and  it  should  be  encouraged  that  ideas  be  offered  that  challenge  conventional  thought. 

The  Cybercraft  pictured  above  originated  as  a  good  idea  from  a  young  officer  in  AFRL. 
He  asked  what  it  would  be  like  to  ‘fly  through  cyberspace’  via  a  capability  analogous  to  an 
aircraft.  This  has  led  to  a  whole  new  way  of  thinking  about  weaponization  in  cyberspace. 

Companies  are  increasingly  using  new  forms  of  internet  software  -  such  as  blogs  and 
wikis  -  to  solicit  ideas  from  throughout  their  organizations,  their  markets,  and  even  from  their 
competitors.  Staging  competitions  to  find  the  best  ideas  is  one  way  to  provoke  new  ideas  that 
challenge  the  status  quo.  For  a  description  of  the  use  of  such  software  in  a  way  that  provokes  or 
encourages  competition,  see  the  recent  book  by  Hagel  and  Brown  (2005). 

One  promising  new  method  of  reaching  a  large  number  and  wide  variety  of  potential 
collaborators  is  through  the  use  of  online  simulations  and  new  forms  of  internet  software.  For 
example,  the  Army  built  a  video  game  called  America’s  Army™  as  a  recruiting  tool.  There  are  a 
huge  number  of  creative  ideas  generated  by  the  online  game  community.  Eight  million  users 
now  play  the  game  online.  A  few  of  the  ideas  have  resulted  in  tangible  benefits  to  the  Army. 
The  game  is  used  in  a  variety  of  real  training  situations  for  Army  recruits.  The  Army  observes 
game  play  to  discover  innovative  tactics.  Picatinny  Arsenal  has  even  included  a  model  of  the 
Stryker  fighting  vehicle  into  the  game  that  suggests  a  novel  way  to  support  OT&E. 
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People 


■  Stand  up  small  ad  hoc  teams  of  exceptional  innovators  to 
conduct  system-level  discovery  experiments 


■  Recruit  and  mentor  innovators 

■  Assemble  Innovation  Task  Forces  (ITFs), 
mixes  of  operators  and  technologists 

■  Collaborate  freely  -  explore  ideas 
as  a  team 

■  Ensure  existence  of  a  champion 


Identify,  recruit,  and  protect  the  innovators! 


Small  teams  of  innovators  who  challenge  the  status  quo 


Small  teams  of  individuals  predisposed  to  innovation  will  execute  system-level 
experiments. 

There  are  people  that  are  predisposed  to  innovation.  It  is  proposed  that  they  brought 
together  in  ad  hoc  teams  (i.e.,  innovation  task  forces)  for  short  periods  of  times  to  support  these 
experiments.  Such  teams  operate  for  finite  period  of  time.  Protection  of  the  team  members  is 
critical.  They  need  the  freedom  to  innovate,  and  this  includes  what  might  be  thought  of  as 
failure  in  other  venues. 

How  does  one  identify  people  predisposed  to  innovation?  Some  good  ideas  are  in  Coates 
(2004)  and  Synder  and  Duarte  (2003).  But  as  pointed  out  by  Natalie  Crawford  (SAB  Senior 
Mentor  and  Director  of  Rand  Corporation’s  Project  Air  Force),  if  appropriate  venues  are 
provided  that  allow  such  experimentation  to  take  place,  the  innovators  “will  flock  to  this  like 
bees  to  honey.”  As  observed  previously,  the  Air  Force  has  many  people  predisposed  to 
innovation.  The  need  is  for  the  leadership  to  sanction  and  support  disruptive  innovation  and  to 
protect  those  few  people  selected  to  engage  in  such  experiments. 
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Venues 


■  Conduct  experiments  in  the  right  venues 


■  Leverage  the  network  of  innovators  in  the  Air  Force 


■  Provide  an  insight-sharing  space  for 
experimentation 

■  Based  on  challenge-competitive 
gaming  environments 

■  Ranges  from  simple  “seminar 
explorations”  to  networked 
reality  games 

■  Allow  for  multiple  layers  of  realism 
and  exploration 


Not  necessarily  a  physical  space;  an  insight-sharing  space 
for  exploration  and  discovery 


Experimentation  venues  are  not  necessarily  physical  spaces,  but  rather  insight-promoting 
spaces  where  effective  exploration  of  ideas  and  discovery  of  hidden  insights  occur.  System-level 
experimentation  venues  are  different  from  those  used  for  training  or  demonstrations.  They  are 
fundamentally  based  on  challenge-competitive  gaming  environments  in  which  the  merits  of  ideas 
for  the  interplay  between  technology  and  CONOPS  can  be  explored.  The  venues  provide  the 
“mixing  bowl”  within  which  operators  and  technologists  interact  to  explore  and  evolve  ideas. 

The  most  productive  venue  must  be  used  for  each  stage  of  an  experiment  campaign,  with 
venues  differing  at  various  stages  of  the  campaign.  Early  stages  of  experimentation,  like  simple 
“seminar  games,”  allow  for  brainstonning  to  provide  an  understanding  of  broad  concepts  and 
allow  rapid  evolution  of  potential  approaches.  Following  stages  may  range  from  “maps  on  the 
floor”  competitions  to  networked  reality  games  that  explore  the  intended  and  unintended  effects 
of  proposed  solutions  in  near-fully  operational  contexts.  Final  stages  of  the  experiment 
campaign  may  include  field  experimentation  or  even  experimental  operational  implementation. 

Networked  reality  games  represent  an  emerging  class  of  venues  that  could  become  a 
major  enabling  element  for  SLE.  Experience  with  such  games  -  from  DARPA’s  early  SIMNET 
environment  to  today’s  MMOGs  -  shows  that  detailed  realism  in  the  venue  itself  is  not  needed  to 
provide  high  fidelity  in  such  games.  More  important  are  a  rapid  pace  of  operations,  the  almost 
limitless  degrees  of  freedom,  and  the  resulting  “fog  of  war”  created  by  interactions  among  large 
numbers  of  players,  and  the  ability  to  work  in  a  highly  collaborative,  competition-based 
environment.  Today  MMOGs,  and  especially  team-based  tactical  first-person  shooter  games 
such  as  Counter-Strike™,  provide  venues  in  which  highly  complex,  collaborative,  competition- 
based,  SLEs  could  be  conducted.  Adding  the  capability  in  such  venues  for  participants  -  both 
friendlies  and  adversaries  -  to  define  their  own  capabilities,  unfettered  by  doctrinal  or  cultural 
limitations  and  bound  only  by  physics,  would  enable  a  virtual  environment  for  true  SLE. 
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Experiments 


■  Learn  from  campaigns  of  genuine  discovery  experiments 


■  Explore  interplay  between  technology  and 
CONOPS 

■  Campaigns  of  discovery  experiments  allow  full 
exploration  of  ideas 

■  Challenge-competition  maximizes  pace  of 
discovery  and  depth  of  exploration 


■  Unfettered  and  highly-skilled  adversary;  no 
cultural  limitations;  physics-only  restrictions 

■  Expect  initial  experiments  to  show  lower 
performance  relative  to  existing  metrics 

■  Capture,  archive  and  mine  experiment  results 
to  develop  insights 


Experiment  with  the  system,  not  just  the  technology 


A  “campaign  of  experiments”  is  the  process  by  which  the  ad  hoc  experimentation  teams 
explore  ideas  for  the  interplay  between  technologies  and  CONOPS,  develop  the  insights  that 
produce  a  deep  understanding  of  potential  future  environments,  and  discover  combinations  that 
provide  innovative  solutions  to  a  problem.  For  any  given  problem,  the  campaign  is  a  sequence 
of  challenge-based  discovery  experiments  that  progresses  from  a  typically  simple  initial  venue  to 
a  final  one  in  which  the  proposed  solution  can  be  understood  in  a  near-fully  operational  context. 

Frequent  experimentation  and  analysis  of  results  are  keys  to  achieving  system-level 
innovations.  Each  experiment  in  the  campaign  is  planned  based  on  the  experience  and  insights 
obtained  from  those  preceding  it.  Results  from  early  experiments  exploring  new  solution 
approaches  may  compare  poorly  when  measured  by  metrics  established  for  currently  accepted 
solution  paths.  Over  the  course  of  the  campaign,  the  succession  of  experiments  explores 
increasingly  deeper  aspects  of  the  problem  to  develop  a  clearer  understanding  of  potential 
solution  approaches.  A  role  of  the  “champion”  is  to  protect  the  experiment  campaign  from 
pressures  to  revert  to  the  status  quo  long  enough  for  the  SLE  process  to  work. 

Capturing  and  archiving  results  from  the  sequence  of  experiments  is  central  to  an 
effective  experimentation  campaign.  This  facilitates  analysis  and  insight  and  allows  design  of 
the  next  step  in  the  campaign.  Data  mining  of  the  experiment  results  permits  hidden  insights 
within  them  to  be  discerned.  For  experiments  involving  massively  multiplayer  online  game 
venues  such  as  Counter-Strike™,  where  200,000  or  more  players  worldwide  may  be  involved  in 
thousands  of  simultaneous  games,  the  amounts  of  observational  data  collected  by  monitoring 
gameplay  can  be  enormous.  Data  mining  can  then  have  potentially  profound  benefits;  the  range 
of  innovations  being  conceived  and  explored  in  such  games  can  far  exceed  what  an  ad  hoc 
experimentation  team  could  possibly  do. 
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Example  SLEs  That  Could 
Change  the  Game 

■  Find  Once,  Track  Forever 
(Non-blinking  Eye  in  the  Sky) 

■  Cockpit  Selectable  Weapon  Effects 

■  Eliminate  Batteries 

■  Directed  Energy  Technology  in  the  Joint  Urban  Fight 

■  Massively  Multiplayer  Online  Games 


Stage  a  competition  of  ideas  and  then  assemble 
Innovation  Task  Forces  (ITFs)  to  execute  the  SLEs 


Sc 

d&f 


Several  potential  SLEs,  the  first  four  were  discussed  with  the  AFSOC  Commander  and 
his  staff,  are  provided  as  examples.  For  each,  one  can  ask  the  following  questions.  How  would 
CONOPS  change?  What  “fog  and  friction  of  war”  issues  would  constrain  this  capability?  What 
technical  issues  would  have  to  be  resolved?  What  series  of  discovery  experiments  should  be 
conducted  to  explore  these  questions? 

Find  Once,  Track  Forever  (Non-blinking  Eye  in  the  Sky).  The  picture  on  the  chart  is 
a  mock  up  Iraqi  town  at  Twenty-Nine  Palms,  California.  Consider  a  new  technical  capability  - 
browseable  video  -  that  allows  an  analyst  or  operator  to  pull  desired  information.  Recent 
technical  experiments  conducted  by  AFRL  and  AFIT  suggest  this  may  be  technically  feasible. 

Cockpit  Selectable  Weapon  Effects.  Ultra  precision  effects  are  desirable  in  urban 
environments.  Suppose  a  “dial  an  effect’  munitions  capability  could  be  specified  as  late  as  near 
impact  of  the  munition. 

Eliminate  Batteries.  Suppose  power  were  not  an  operational  consideration  for 
Battlefield  Airmen?  These  combat  controllers,  pararescuemen,  and  combat  weathermen  often 
carry  packs  weighing  several  hundred  pounds  while  executing  dangerous  and  covert  missions  in 
austere  locations  around  the  world.  Thirty  percent  of  the  weight  is  batteries. 

Directed  Energy  Technology  in  the  Joint  Urban  Fight.  Another  example  might  be 
directed  energy  technology  in  the  joint  urban  fight.  Despite  power  constraints,  what  if  airborne 
DE  capabilities  were  fielded? 

Massively  Multiplayer  Online  Games.  America’s  Army™  today  has  over  7.2  million 
users;  with  4.2  million  having  the  competency  required  for  full  participation  in  over  2.1  million 
games  (“experiments”)  played  each  day  on  nearly  2,200  game  servers  spread  around  the  globe. 
The  Air  Force  should  create  a  similar  game  to  mine  the  world  for  new  ways  of  thinking. 

Public  Release 
33 


Public  Release 


In  Summary,  Key  Points 


■  Ideas  -  from  everywhere,  challenge  conventional  thought 

■  Protect  innovators 

■  Experiment! 

■  Conduct  a  campaign  of  discovery  experiments 

■  Do  it  outside  core  processes 

■  Focus  on  co-evolution  of  technology  and  CONOPs 

■  Stress  via  an  unconstrained  adversary 

■  Failure  is  OK;  Learning  is  the  objective 

■  Leadership  in  innovation  must  start  with  the  CEO 


The  Air  Force  must  again  become  a  leader  in 
disruptive  innovation 


The  four  components  of  SLEs  are  listed  below.  They  are  a  combination  of  recently 
codified  industry  best  practices  for  disruptive  innovation  and  the  insight  of  this  study  with 
respect  to  the  role  of  early  discovery  experimentation  focused  on  system  alternatives  in  a  future 
environment. 

•  Ideas  -  cast  the  net  wide;  challenge  all  existing  doctrinal  limits 

•  People  -  exceptional  innovators;  driven  to  challenge  convention 

•  Venue  -  a  challenge-competitive  exploration  space 

•  Experiments  -  iterative  campaigns  of  discovery  experiments 

These  components  are  based  on  recently  codified  best  practices  for  innovation  in  the 
commercial  world  and  the  historic  role  of  experimentation  in  the  Air  Force. 
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Organizing  for  Innovation 


AF  Chief  of  Innovation 


System-Level  Experimentation 


The  graphic  suggest  how  to  organize  for  SLEs.  Ideas  need  to  be  solicited  from 
throughout  the  Air  Force  and  the  external  community.  A  coordinating  office,  headed  by  the 
Chief  of  Innovation,  selects  the  best  ideas  for  experimentation  and  selects  innovators  to  serve  on 
Innovation  Task  Forces  (ITFs).  The  ITFs  conduct  the  SLEs  in  an  appropriate  venue  -  either  in 
an  operational  lab  location  (but  not  in  facilities  owned  by  the  coordinating  office).  The 
coordinating  office  mentors,  facilitates,  and  ensures  the  capture  of  lessons  learned  of  the 
experiments.  In  addition,  the  coordinating  office  facilitates  the  transition  of  experimental 
outcomes  to  the  appropriate  Air  Force  functions  (e.g.,  ACC/A8,  inclusion  in  an  out  year  JEFX, 
incorporation  into  an  AFRL  program,  rapid  fielding  via  RCO,  etc.).  It  is  critical  that  the  Chief  of 
Innovation  and  the  coordinating  office  for  SLEs  operate  outside  core  processes  for  requirements 
definition  and  acquisition.  It  is  equally  critical  that  the  Chief  of  Innovation  report  to  the  CSAF 
and  that  he  or  she  have  sufficient  resources  and  authority  to  conduct  and  pursue  disruptive 
innovation. 
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Recommendations 


■  Identify  Chief  of  Innovation 

■  Selected  by  and  reports  to  CSAF 

■  Must  be  predisposed  to  disruptive  innovation 

■  Serves  as  idea  provocateur  and  harvester 

■  Must  have  authority  and  resources 

■  Support  these  functions 

■  Coordinate  across  the  Air  Force  to  identify  SLE  opportunities 

■  Assemble  Innovation  Task  Forces  (ITFs)  to  execute  SLEs 

■  Evaluate  SLE  results;  ensure  they  are  captured,  shared  and  used 

■  Assess  and  facilitate  transition  opportunities 


Create  a  climate  that  encourages  and  rewards  innovation 


The  recommendation  follows  from  what  is  being  quickly  adopted  in  the  commercial 
world  -  the  designation  of  a  Chief  of  Innovation  with  a  direct  reporting  line  to  the  CEO.  Hence, 
it  is  recommended  that  the  CSAF  appoint  someone  with  a  direct  reporting  line  to  serve  this 
function.  The  person  must  be  tasked  and  predisposed  to  pursue  disruptive  innovation  outside  the 
core  processes  of  requirements  definitions  and  acquisition.  This  person  needs  to  be  provided 
sufficient  resources  and  authority  to  solicit  ideas  widely  throughout  the  Air  Force  and  its  external 
community  and  to  pursue  three  to  four  SFEs  per  year. 


Public  Release 
36 


Public  Release 


So  when  someone  asks  you  what  the 
Air  Force  will  be  doing  in  the  future,  tell  them 
this.  ...  We  will  innovate,  and  we  will  fly,  we 

will  fight,  and  we  will  win. 

General  T.  Michael  Moseley,  Sep  2005 


This  is  a  quote  by  Air  Force  Chief  of  Staff  General  T.  Michael  Moseley  to  the  Air  Force 
Association  Air  and  Space  Conference  on  14  September  2005. 
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Appendix  A:  Terms  of  Reference 


USAF  Scientific  Advisory  Board 
Quick  Look  Study 
FY  2006 

System-Level  Experimentation  in  Air  Force  S&T  Programs 

Terms  of  Reference 

Background 

An  approach  to  experimentation  in  today’s  rapidly  changing  threat  environment  is  needed  to 
concurrently  support  the  discovery  and  creation  of  new  operational  and  system  concepts  and  the 
technology  needed  to  support  those  concepts.  Such  an  approach  would  encourage  creative 
exploration  of  the  interplay  among  technology,  system,  and  operational  concepts  and  help 
establish  a  deep  understanding  of  a  future  environment.  While  the  focus  would  be  on  evolving 
S&T,  the  approach  must  include  frequent  technologist-operator  interaction,  accommodate  the 
“fog  and  friction  of  war”  (e.g.,  an  adversarial  component,  system  integration  issues),  involve 
higher  risk  than  typically  allowed  in  demonstration-based  exercises  (e.g.,  JEFX),  and  be 
implementable  within  technology  transition  processes. 

Study  Products 

Briefing  to  SAF/OS  &  AF/CC  in  June  2006.  Publish  report  in  August  2006. 

Charter 

The  “quick  look”  study  will  propose  an  approach  to  system-level  experimentation  in  Air  Force 
S&T  programs  by  providing  the  following: 

•  An  overview  of  current  Air  Force  “state  of  practice”  and  published  approaches  for 
exploratory  experiments  involving  technologists  and  operators. 

•  An  overview  of  approaches  practiced  by  others  (e.g.,  other  Services,  national 
laboratories,  commercial  industry,  adversaries). 

•  A  description  of  an  approach  by  which  system-level  experiments  could  be  used  to  ‘stress 
test  technology,’  focus  high  payoff/high  risk  research,  and  envision  and  understand  new 
operational  and  system  concepts. 

•  A  description  of  three  system-level  experiments  in  the  domain  of  Air  Force  special 
operations. 

•  Recommendations  for  a  potential  follow-on  study  to  address  the  scale-up  of  the  approach 
and  enhancements  to  technology  transition  processes  to  fully  realize  the  benefits. 
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Appendix  B:  Study  Members 


Study  Leadership 

Dr.  Stephen  Cross,*  Co-Chair  Georgia  Institute  of  Technology 
Mr.  Scott  Fouse,*  Co-Chair  ISX  Corporation 


Study  Members 

Dr.  Robert  Byer* 

Prof.  Werner  Dahm* 

Dr.  Ken  Ford* 

Dr.  Richard  LeSar* 

Maj  Gen  Eric  Nelson,  USAF  (Ret) 
Dr.  Brad  Parkinson* 

Mr.  Skip  Saunders* 


Stanford  University 

University  of  Michigan 

Institute  for  Human  and  Machine  Cognition 

Los  Alamos  National  Laboratory 

Independent  Consultant 

Stanford  University 

Independent  Consultant 


Study  Management  and  Support 

Major  Valerie  Manning,  USAFR  AF/SB  -  Program  Manager 
Major  Lee  Chase,  AFRL/PR  -  Executive  Officer 
Captain  Justin  Joffrion,  USAFA  -  Executive  Officer 
Major  Matt  Keihl,  AFRL/DE  -  Technical  Writer 
Mr.  Justin  Waters,  AF/SB  -  Analyst 


*Denotes  current  status  as  a  member  of  the  Air  Force  Scientific  Advisory  Board. 
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Appendix  C:  Visits  and  Briefings 


Air  Force 

Secretary  of  the  Air  Force 

Assistant  Secretary  of  the  Air  Force  for  Acquisition 

•  Deputy  Assistant  Secretary  of  the  Air  Force  for  Science,  Technology,  and  Engineering 

•  Rapid  Capabilities  Office 

Deputy  Chief  of  Staff  for  Strategic  Plans  and  Programs 

•  Future  Concepts  Development 

Chief  of  Warfighting  Integration  and  Chief  Information  Officer,  Office  of  the  Secretary  of  the 
Air  Force 

•  Strategy  and  Plans 

•  Innovation  and  Technology 
Air  Mobility  Command 

•  Strategic  Plans  and  Programs 
Air  Force  Special  Operations  Command 

•  Commander 

•  Intelligence 

•  Air,  Space,  and  Infonnation  Operations 

•  Logistics 

•  Plans,  Programs,  Requirements,  and  Assessments 

•  Communications  and  Information 
Air  Force  Materiel  Command 

•  Strategic  Plans  and  Programs 
Air  Force  Space  and  Missile  Center 

•  Vice  Commander 

•  MILSATCOM  Joint  Program  Office 
Air  Force  Research  Laboratory 

•  Commander 

•  Chief  Technologist 

•  Air  Vehicles  Directorate 

•  Directed  Energy  Directorate 

•  Human  Effectiveness  Directorate 

•  Propulsion  Directorate 

•  Sensors  Directorate 

•  Space  Vehicles  Directorate 
Air  Force  C2ISR  Center 

Air  Warfare  Battlelab 
Command  and  Control  Battlelab 
Unmanned  Aerial  Vehicle  Battlelab 
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Air  Force  continued 

Air  University 

•  Air  Command  &  Staff  College 

•  Air  War  College 

o  Center  for  Strategy  &  Technology 

•  School  for  Advanced  Air  and  Space  Studies 
U.S.  Air  Force  Academy 

Other  Government  /  FFRDCs 

U.S.  Joint  Forces  Command 

•  Joint  Experimentation  Directorate 

•  Joint  Forces  Intelligence  Command 
U.S.  Army  Training  and  Doctrine  Command 
U.S.  Navy  Warfare  Development  Command 

•  Commander 

•  Chief  Engineer 
Office  of  Naval  Research 

•  Chief  Scientist 
U.S.  Marine  Corp 
NASA 

Los  Alamos  National  Laboratory 
Sandia  National  Laboratories 

Industry 

Cisco 

Council  on  Competitiveness 
IBM 

Institute  for  Human  &  Machine  Cognition 


Public  Release 
44 


Public  Release 


Appendix  D: 

ACC/A8 

AF 

AF/A8 

AFIT 

AFRL 

AFSOC 

ARS 

ATD 

AU 

AU/CC 

AWACS 

BA 

BAO 

CEO 

COI 

CONOPs 

CSAF 

CSAT 

DARPA 

DDR&E 

DE 

DMOC 

DMT 

FBE 

GPS 

IADS 

ICBM 

ICE 

IED 


Acronyms  and  Abbreviations 


Air  Combat  Command  Director  of  Requirements 
Air  Force 

Deputy  Chief  of  Staff,  Strategic  Plans  and  Programs 

Air  Force  Institute  of  Technology 

Air  Force  Research  Laboratory 

Air  Force  Special  Operations  Command 

Advanced  Reconnaissance  System 

Advanced  Technology  Demonstration 

Air  University 

Air  University  Commander 

Airborne  Warning  and  Control  System 

Battlefield  Airman 

Battlefield  Airman  Operations 

Chief  Executive  Officer 

Chief  of  Innovation 

Concept  of  Operations 

Air  Force  Chief  of  Staff 

Center  for  Strategy  and  Technology 

Defense  Advanced  Research  Projects  Agency 

Director  of  Defense  Research  and  Engineering 

Directed  Energy 

Distributed  Mission  Operations  Center 
Distributed  Mission  Training 
Fleet  Battle  Experiment 
Global  Positioning  System 
Integrated  Air  Defense  System 
Intercontinental  Ballistic  Missile 
Integrated  Collaborative  Environment 
Improvised  Explosive  Device 
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IPA 

Intergovernmental  Personnel  Act 

IRBM 

Intermediate-range  Ballistic  Missile 

ITF 

Innovation  Task  Force 

JDAM 

Joint  Direct  Attack  Munition 

JEFX 

Joint  Expeditionary  Force  Experiment 

JSOW 

Joint  Stand-Off  Weapon 

KPP 

Key  Performance  Parameter 

MMOG 

Massively  Multiplayer  Online  Games 

OT&E 

Operational  Test  and  Evaluation 

PC 

Personal  Computer 

R&D 

Research  and  Development 

RATSCAT 

Radar  Target  Scatter 

RCO 

Rapid  Capabilities  Office 

RCS 

Radar  Cross  Section 

ROE 

Rules  of  Engagement 

S&T 

Science  and  Technology 

SAB 

Scientific  Advisory  Board 

SAMOS 

Satellite  and  Missile  Observation  Satellite 

SIMNET 

SIMulation  NET  work 

SLE 

System-Level  Experimentation 

TOR 

Terms  of  Reference 

U.S. 

United  States 

UAV 

Unmanned  Aerial  Vehicle 

USAF 

United  States  Air  Force 

VLO 

Very  Low  Observable 

XST 

Experimental  Survivable  Testbed 
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Appendix  E:  Case  Histories  in  Disruptive  Innovation 


Military  history  contains  many  examples  of  disruptive  innovation.  Each  of  these 
disruptive  innovations  relied  partially  on  technology  and  partially  on  the  creative  insights  shared 
between  operators  and  technologists  on  how  best  to  use  that  technology.  The  military  did  not 
develop  these  solutions  as  products  of  a  formal  acquisition  process;  rather,  innovators 
incorporated  technologies  and  ideas  then  experimented  with  tactics  and  processes  to  change  the 
conduct  of  war.  This  study  looked  at  a  number  of  these  historical  examples  and  for  each  one  has 
summarized  the  essential  elements  that  enabled  the  disruptive  innovation. 

The  following  examples  of  disruptive  innovation  are  included  in  this  section: 


Sinking  of  the  Ostfriedland  -  Mitchell  (1921) 
Carrier  Warfare  -  US  Navy  (1930s) 

Early  Warning  System  -  Chennault  (1933) 
Blitzkrieg  -  Guderien  (1940s) 

ICBM  -  Schriever  (1950s) 

Space-based  Surveillance  -  (1960s) 

Precision  Guided  Weapons  -  (1970s) 

Stealth -(1970s) 

Armed  Predator  -  (2002) 
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Sinking  of  the  Ostfriedland 

Gen  Billy  Mitchell  is  well  known  as  one  of  the  Air  Force’s  great  innovators.  Fie 
achieved  this  acclaim  in  1921  when  he  set  out  to  prove  that  no  naval  fleet  could  survive  an  attack 
by  land-based  air. 

...Mitchell  stated  that  1,000  bomber  aircraft  could  be  built  and  operated  for  the  cost  of  one 
dreadnought  and  that  his  airplanes  could  sink  a  battleship.  He  volunteered  to  demonstrate 
this  if  the  navy  would  provide  him  with  some  battleships,  which  were  already  due  to  be 
demolished.  The  navy  reluctantly  agreed  to  the  demonstrations. 

Once  it  was  decided  the  tests  would  be  performed  under  navy  rules,  it  set  strict  guidelines 
that  it  felt  would  ensure  failure  for  Mitchell.  One  was  that  the  bombings  be  conducted 
slowly,  stopping  often  to  permit  inspections  of  the  damage  by  construction  inspectors.  This 
would  allow  a  scientific  appraisal  of  the  capacity  of  different  types  of  ships  to  withstand 
aerial  attack.  Also,  the  number  and  size  of  the  bombs  were  to  be  limited. 

The  tests  began  in  July  off  the  coast  of  Virginia.  The  navy  had  provided  Mitchell  with 
three  decommissioned  U.S.  battleships  and  three  ships  obtained  from  the  Germans  in  the 
peace  agreement— a  destroyer,  an  armored  light  cruiser,  and  a  dreadnought.  All  were 
successfully  sunk.  The  climax  of  the  demonstrations  took  place  on  July  21,  when  the  navy 
brought  out  the  German  ship  Ostfriedland,  a  great  ship  that  had  been  the  pride  of  the 
German  fleet  during  the  war.  The  vessel  was  considered  unsinkable,  and  it  probably  would 
have  been  if  Mitchell  had  adhered  to  the  rules.  But  instead,  he  had  personally  overseen  the 
design  of  a  number  of  2,000-pound  ( 907-kilogram )  bombs,  knowing  that  smaller  bombs 
would  not  be  successful.  Martin  twin-engine  MB-2  bombers  dropped  six  of  these  bombs  in 
rapid  succession.  Two  scored  direct  hits  and  the  others  landed  close  enough  for  the  ship ’s 
hull  plates  to  rip  open  from  the  force  of  the  explosion.  Twenty-one  minutes  after  the  test 
began,  the  Ostfriedland  plunged  to  the  bottom  of  the  ocean.  The  final  plane  dropped  its 
bombs  into  the  foam  rising  from  the  sinking  ship. 

The  navy  was  horrified  and  declared  the  tests  void  since  Mitchell  had  violated  the 
guidelines.  But  it  also  began  to  focus  more  on  aviation.  The  Bureau  of  Aeronautics,  which 
had  been  established  in  1921  as  a  defense  against  Mitchell’s  actions  under  the  leadership 
of  William  Moffat,  increased  its  development  of  the  aircraft  carriers  that  would  eventually 
help  win  the  Pacific  campaign  in  World  War  II. 

Mitchell’s  experiment  is  a  prime  example  of  an  unfettered  adversary.  If  Mitchell  had 
adhered  to  the  Navy’s  restrictions,  then  he  would  have  not  been  able  to  fully  demonstrate  the 
supremacy  of  air  power.  Mitchell’s  experiment  identified  a  major  vulnerability  in  the  Navy’s 
fleet,  and  they  eventually  altered  their  strategic  plans  to  develop  aircraft  carriers  instead  of  the 
dreadnought. 


Ref: 

Feltus,  P.  “Billy  Mitchell  Sinks  the  Ships.”  U.S.  Centennial  of  Flight  Commission. 
http://www.centennialofflight.gov/essav/Air  Power/mitchell  tests/ AP  14.htm. 
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Carrier  Warfare  (The  Early  Development  of  Naval  Aviation) 

An  overview  of  experimentation  in  the  creation  of  the  Navy’s  Aircraft  Carrier  capability 
is  in  a  recent  National  Academy  of  Engineering  report.  A  section  is  excerpted  here. 

The  history >  of  the  Navy ’s  use  of  experimentation  to  achieve  new  capabilities  is  illustrated 
by  the  role  of  experimentation  in  the  introduction  of  aircraft  and  aircraft  carriers.  The 
motivation  to  undertake  an  experimentation  campaign  related  to  naval  aviation  was  driven 
directly  by  a  decision  of  Admiral  of  the  Fleet  George  Dewey,  who  began  to  push  the 
concept  of  naval  aviation  after  viewing  the  use  of  dirigibles.  The  admiral  is  said  to  have 
commented,  “If  you  can  fly  higher  than  the  crow’s  nest,  we  will  use  you.  ”  To  pursue  the 
concept  of  naval  aviation,  Captain  Washington  Chambers  was  designated  by  Admiral 
Dewey  as  the  Navy ’s  lead  aviation  project  officer.  Chambers ’s  jobs  were  to  find  funding  for 
the  project  and  to  demonstrate  that  an  aircraft  could  both  take  off  from  and  land  on  a  ship. 

George  von  L.  Meyer,  then  Secretary >  of  the  Navy,  refused  to  include  funds  in  the  budget  for 
the  demonstration.  Not  to  be  deterred,  Chambers  found  a  rich,  politically  well  connected 
publisher  and  aviation  enthusiast  named  John  B.  Ryan  to  help  him.  Ryan  contacted 
President  Taft,  who  persuaded  Secretary  Meyer  to  change  his  mind  and  designate  the 
cruiser  USS  Birmingham  to  be  used  for  the  experiment.  The  experiment  required  the 
construction  of  a  wooden  ramp  extending  from  bridge  to  bow.  While  the  Navy  provided  the 
ship,  the  cost  of  the  ramp  ($288)  was  paid  for  by  Ryan.  The  first  demonstration  of  an 
aircraft  taking  off  from  a  ship  took  place  near  Norfolk,  Virginia,  in  November  1910. 

Captain  Chambers  was  then  authorized  to  spend  not  more  than  $500  to  construct  an 
aircraft  recovery >  ramp  on  the  stern  of  the  cruiser  USS  Pennsylvania.  On  the  basis  of 
experiments  ashore,  Chambers  and  his  pilot,  Eugene  Ely,  determined  that  arresting  cables 
would  be  needed  to  bring  the  aircraft  to  a  stop.  Accordingly,  15  cables  were  stretched 
across  the  deck,  each  fastened  at  either  end  to  a  50-lb  sandbag. 

In  1913,  Captain  Chambers  determined  that  all  available  aircraft  and  pilots  should  take 
part  in  the  fleet’s  winter  exercises  of  1913  off  Guantanamo  Bay,  Cuba.  These  annual 
exercises  were  the  equivalent  of  the  current  Navy  fleet  battle  experiments  (FBEs).  For  these 
experiments,  Chambers ’s  officers  rigged  a  wireless  transmitter  on  one  of  the  aircraft  and  a 
receiver  on  the  flagship.  An  aircraft  then  flew  over  the  horizon  to  scout  out  the  position  of 
the  opposing  forces.  Although  transmission  took  place  on  the  plane,  reception  on  the 
flagship  did  not  occur.  However,  the  concept  of  using  an  elevated  platform  to  locate  hostile 
forces  had  been  established. 

During  the  next  7  years,  aircraft  technology ’ — driven  by  needs  of  the  Allied  and  Central 
powers  in  World  War  I — accelerated  rapidly,  as  did  the  number  of  qualified  flyers  and 
aircraft  in  the  U.S.  Navy.  By  the  end  of  World  War  I,  aircraft  carried  weapons  (machine 
guns),  could  drop  bombs,  and  could  undertake  primitive  communications.  Experiments  had 
resulted  in  the  development  of  moderately  safe  catapults  that  allowed  pontoon  aircraft  to 
be  launched  from  a  ship’s  fantail.  In  1917,  the  British  Navy  undertook  experiments  with 
arresting  cables  that  could  absorb  the  energy >  of  a  landing  aircraft  more  efficiently  than 
could  Ely ’s  arrangement  of  cables  and  sandbags.  Thus,  by  the  end  of  World  War  I,  all  of 
the  technology >  required  for  an  aircraft  carrier  was  in  place. 

On  March  20,  1922,  the  USS  Langley,  the  Navy ’s  first  aircraft  carrier,  was  commissioned. 

The  ship  had  been  converted  from  the  former  Jupiter,  a  collier.  By  the  end  of  the  decade, 
two  more  carriers,  the  Lexington  and  the  Saratoga,  were  commissioned.  The  performance 

Public  Release 
49 


Public  Release 


of  carrier  aviation  in  the  war  games  (FBEs)  of  1929  was  a  portent  of  the  future.  Opposing 
fleets  were  charged  with  the  attack  and  defense  of  the  Panama  Canal.  The  Saratoga 
(attacking  force),  under  cover  of  darkness  and  bad  weather,  launched  69  aircraft,  which 
arrived  over  and  theoretically  destroyed  the  canal  without  incident.  Thus,  the  role  of  the 
fast  carrier  was  predicted  12  years  before  Pearl  Harbor. 


Ref: 

Naval  Studies  Board.  (2004).  The  Role  of  Experimentation  in  Building  Future  Naval  Forces. 
Washington  DC:  The  National  Academies  Press.  ISBN:  0309088739. 
www.nap.edu/books/0309088739/html/28.html 
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Early  Warning  System 

The  Air  Corps  Tactical  School  at  Maxwell  Field  was  the  heart  of  early  Air  Force 
innovation  in  the  1930s.  An  excellent  example  of  an  early  SLE  is  described  in  Perry’s  paper. 
The  following  illustrates  one  set  of  SLEs  involving  an  early  warning  network  to  provide  an 
unfair  competitive  advantage  to  technically  less  capable  pursuit  aircraft  of  the  day  over  the 
bomber  force. 

Captain  Chennault  “talked  so  loud  and  long  about  the  necessity  for  an  aircraft  warning 
net,  and  providing  radio  intelligence  to  the  defending  fighters  in  the  air,  that  another  air 
force  maneuver  was  held  in  1933  at  Fort  Knox,  Kentucky >.  ”  It  tested  his  proposed  air 
defense  warning  system.  A  line  running  between  Indianapolis  and  Cincinnati  divided  the 
friendly  forces  based  at  Dayton  and  opposition  forces  located  at  Fort  Knox.  The  former 
forces  flew  fast,  modern  bombers  while  the  latter  had  slow,  fabric-covered  biplanes.  Three 
regiments  of  antiaircraft  artillery’  supplied  guns,  searchlights,  and  observers. 

Chennault ’s  warning  system  represented  the  heart  of  the  exercise.  It  covered  a  120  degree¬ 
wide  sector  centered  on  Fort  Knox  and  radiating  out  towards  Dayton  with  69  observation 
posts  at  regular  intervals.  When  planes  were  spotted,  they  telephoned  fighter  control  at 
Fort  Knox  with  the  number,  altitude,  and  course  of  the  aircraft  using  a  simple  three-word 
code.  This  information  was  then  plotted  on  a  map.  Opposition  obsen’ation  planes  circled 
over  the  friendly  base  at  Dayton,  which  had  no  defenses.  These  planes  relayed  their 
intelligence  through  a  radio-equipped  transport  near  Cincinnati.  Prior  experience  had 
shown  that  fighter  control  must  receive  messages  within  four  minutes  or  pursuit  would  not 
be  able  to  intercept.  In  this  exercise,  however,  almost  1,000  messages  were  sent  in  an 
average  time  of  2. 7  minutes.  The  opposition  pursuit  group  commander  kept  his  planes  on 
strip  alert  and  issued  the  scramble  order  via  a  public  address  system.  Information  was 
updated  by  radio  while  the  fighters  were  in  the  air.  Clear,  fast,  precise  reporting  enabled 
the  opposition  to  intercept  friendly  forces  by  day  and  night  and  at  all  altitudes.  Most 
interceptions  occurred  between  25  and  50  miles  from  Fort  Knox,  and  some  bombers  were 
intercepted  more  than  once  per  mission. 

The  exercise  illustrates  the  key  points  of  SLE.  Ideas  were  solicited  from  members  of  the 
Air  Corps  Tactical  School.  The  champion  for  the  ideas  was  Claire  Chennault.  Clearly  he  and 
his  team  were  predisposed  to  disruptive  innovation,  even  at  the  sake  of  their  careers.  The  venue 
was  a  planned  training  exercise.  The  experiments  explored  innovative  ideas  both  with  existing 
technology  and  CONOPS.  Chennault  explored,  in  experiments,  the  utility  of  intelligence  and 
(visual)  early  warning  systems  to  demonstrate  their  effectiveness.  The  but  lessons  not 
assimilated  prior  to  World  War  II. 


Ref: 

Perry,  J.  (1999).  “Air  Corps  Experimentation  in  the  Interwar  years  -  A  Case  Study.”  Joint  Force 
Quarterly,  http://www.dtic.mil/doctrine/iel/ifq  pubs/0922.pdf. 
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Blitzkrieg 

Blitzkrieg,  which  means  "lightening  war,"  was  first  used  by  the  Germans  in  World 
War  II.  It  is  a  military  tactic  based  on  speed,  surprise,  and  constant  advance  -  using  a  force 
composed  of  light  tank  units  supported  by  aircraft,  motorized  infantry,  and  engineers  (to  allow 
the  vehicles  to  cross  rivers  and  other  difficult  terrain  quickly).  The  tactic  was  developed  by  a 
German  officer  named  Heinz  Guderian,  in  whom  Hitler  placed  great  trust  and  authority,  and  who 
implemented  it  very  effectively  in  1939-1941  -  rapidly  defeating  and  occupying  Poland  and 
France,  and  making  the  initial  deep  incursions  into  the  Soviet  Union.  The  combination  of  speed, 
force,  and  technology  resulted  in  a  disruptive  new  way  to  advance  the  German  occupation.  The 
following  excerpt  describes  some  of  the  German’s  process  to  create  and  improve  Blitzkrieg: 

General  Hans  Von  Seeckt  clearly  articulated  that  the  goal  of  the  German  Army  was  a 
return  to  movement  in  hopes  of  avoiding  the  stagnant  trench  warfare  of  World  War  I.  The 
model  emphasized  was:  “offensive,  combined  arms  maneuver,  with  independent  action  by 
officers,  and  intelligent,  effective  leadership  at  all  levels.  ”  Information  was  gathered 
through  the  57  committees  and  validated  through  numerous  training  exercises  and  leader 
development  programs.  Decisions  were  made  using  both  a  combination  of  experiments,  for 
example,  field  testing  new  equipment  of  a  select  unit  rather  than  the  entire  army,  and  using 
systems  already  in  place.  Effects  were  reviewed  in  an  open  forum,  such  as  the  Militdr- 
Wochenblatt,  that  the  military  culture  encouraged.  The  cycle  was  not  as  linear  as  this 
description  might  suggest.  The  process  and  goals  were  constantly  adjusted  as  new 
information  became  available  and,  more  importantly,  as  new  technologies  were  introduced. 

A  perfect  example  was  the  development  of  the  panzer  division.  ...  The  panzer  division  was 
so  effective  that  during  and  after  World  War  II  it  became  a  model  for  many  armies, 
including  France,  Britain,  and  the  United  States. 

The  Germans  effectively  utilized  channels  of  communication  in  order  to  maximize  the 
flow  of  information.  This  information  was  used  throughout  the  experimentation  campaign  to 
make  decisions  and  assess  the  effectiveness  of  the  experimentation.  They  constantly  adapted  as 
new  information  and  new  technology  came  available. 

In  1940,  Britain  and  France  still  had  a  World  War  I  mentality.  What  tanks  they  had  were 
poor  compared  to  the  German  Panzers.  British  and  French  tactics  were  outdated  and  Britain  still 
had  the  mentality  that  as  an  island  we  were  safe  as  our  navy  would  protect  us.  Nazi  Germany,  if 
it  was  to  fulfill  Hitler’s  wishes,  had  to  have  a  modem  military  tactic  if  it  was  to  conquer  Europe 
and  give  to  Germany  the  “living  space”  that  Hitler  deemed  was  necessary  for  the  Third  Reich. 


Ref: 

Gukeisen,  T.  (2005).  “The  Operational  Art  of  Blitzkrieg:  Its  Strengths  and  Weaknesses  in  Systems 
Perspective.  U.S.  Army  Command  and  General  Staff  College  School  of  Advanced  Military  Studies.  DTIC 
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ICBM 


During  and  after  World  War  II,  Ballistic  missile  programs  had  a  low  priority  within  the 
Air  Force,  which  instead  concentrated  its  resources  on  manned  bombers  and  jet  fighters  which 
were  needed  for  the  Korean  conflict.  Most  people  believed  that  a  practical  ICBM  was  far  off, 
due  to  the  weight  of  the  warhead.  That  belief  changed  in  1953  when  Princeton  mathematician 
John  von  Neumann  gave  a  presentation  to  the  SAB  that  indicated  a  1500  warhead  capable  of  one 
megaton-yield  could  be  developed  by  the  end  of  that  decade.  This  realization  immediately  raised 
the  priority  for  ICBM  to  number  one  and  then  the  challenge  was  how  to  quickly  respond  to  this 
high  priority  challenge. 

The  ICBM  development  history,  which  is  well  documented  in  the  book  The  United  States 
Air  Force  and  the  Culture  of  Innovation  by  Stephen  B.  Johnson,  is  a  very  good  example  of  the 
kind  of  innovation  that  can  occur  when  you  have  the  right  mix  of  people  and  ideas. 

In  1954  AF  Secretary  Harold  E.  Talbott  formed  the  Strategic  Missiles  Evaluation 
Committee,  or  “Teapot  Committee,”  to  investigate  and  recommend  a  course  of  action  for 
strategic  ballistic  missiles.  They  recommended  creating  an  organization  that  harkened  back  to 
the  Manhattan  Project  and  MIT’s  Radiation  Laboratory  of  World  War  II. 

The  nature  of  the  task  for  this  new  agency  requires  that  overall  technical  direction  be  in 
the  hands  of  an  unusually  competent  group  of  scientists  and  engineers  capable  of  making 
systems  analyses,  supervising  the  research  phases,  and  completely  controlling  the  experimental 
and  hardware  phases  of  the  program  -  the  present  ones  as  well  as  the  subsequent  ones  that  will 
have  to  be  initiated. 

Gen  Shriever  was  the  champion  for  this  incredible  effort,  and  he  was  able  to  enlist 
support  from  some  of  the  greatest  minds  around,  including  John  von  Neumann,  who  was  the 
head  of  the  SAB’s  Nuclear  Weapons  Panel  and  became  the  head  of  the  Scientific  Advisory 
Committee  for  the  ICBM  project.  Gen  Shriever  successfully  found  new  methods  of  operation, 
cut  through  red  tape,  and  circumvented  meddling.  In  addition,  he  was  given  broad  authority  to 
manage  the  program  as  he  saw  fit.  Shriever’ s  team  overcame  several  significant  technical 
challenges  dealing  with  propulsion,  guidance,  and  re-entry  among  others. 

Gen  Schriever  did  what  many  considered  impossible:  He  had  led  the  development  of  four 
major  missile  systems  -  Atlas,  Thor,  Titan  and  Minuteman  I  -  overseen  the  employment  of 
roughly  300,000  people,  administered  a  budget  for  these  projects  of  nearly  $17  billion,  and 
brought  them  to  Initial  Operational  Capability  in  seven  years.  It  transformed  the  USAF  and  the 
nation  and  helped  usher  in  the  nuclear  deterrence  capabilities  which  were  the  basis  of  national 
security  throughout  the  remainder  of  the  cold  war. 


Ref: 

Johnson,  S.  (2002j.  The  United  States  Air  Force  and  the  Culture  of  Innovation.  Washington 
D.C.:  Air  Force  History  and  Museums  Program. 
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Space-based  Surveillance 

March  16,  1955,  the  Air  Force  issued  General  Operational  Requirement  No.  80,  officially 
establishing  a  high-level  requirement  for  an  advanced  reconnaissance  satellite.  The  document 
defined  the  Air  Force  objective  to  be  the  provision  of  continuous  surveillance  of  “preselected 
areas  of  the  earth”  in  order  “to  detennine  the  status  of  a  potential  enemy's  warmaking 
capability.” 

Over  the  next  five  years  the  U.S.  reconnaissance  satellite  program  evolved  in  a  variety  of 
ways.  The  success  of  the  Soviet  Union’s  Sputnik  I  and  II  satellites  in  the  fall  of  1957  provided  a 
spur  to  all  U.S.  space  programs  including,  reconnaissance.  The  Air  Force  program  was  first 
designated  the  Advanced  Reconnaissance  System  (ARS),  then  SENTRY,  and  finally  SAMOS. 
The  main  objective  of  SAMOS  was  to  develop  a  satellite  that  could  scan  its  exposed  film  and 
return  the  imagery  electronically. 

In  the  early  1960s,  CORONA  replaced  SAMOS  as  the  primary  USAF  reconnaissance 
program.  It  took  over  a  year,  starting  in  1959,  and  14  launches  before  an  operational  CORONA 
spacecraft  was  placed  in  orbit.  The  thirteenth  mission,  a  diagnostic  flight  without  camera 
equipment,  was  the  first  success  -  then  on  the  fourteenth  mission  (August  18,  1960),  a  camera- 
outfitted  CORONA  was  placed  into  orbit  for  a  day.  This  flight  yielded  more  images  of  the 
Soviet  Union  in  its  single  day  of  operation  than  did  the  entire  U-2  program.  The  CORONA 
program  allowed  the  U.S.  to  update  the  estimate  of  the  Soviet  ICBM  force  to  between  25  and  50. 
CORONA  imagery  also  allowed  the  U.S.  to  catalog  Soviet  air  defense  and  anti-ballistic  missile 
sites,  nuclear  weapons  related  facilities,  submarine  bases,  IRBM  sites,  airbases  -  as  well  as 
Chinese,  East  European,  and  other  nations’  military  facilities.  It  also  allowed  assessment  of 
military  conflicts  -  such  as  the  1967  Six-Day  War  -  and  monitoring  of  Soviet  arms  control 
compliance.  Following  the  CORONA  program,  U.S.  space-based  surveillance  capabilities 
continued  to  develop,  leading  to  a  full  space  imaging  fleet  employed  today. 


Ref: 

Richelson,  J.  (1999).  “US  Satellite  Imagery,  1960-1999.”  National  Security  Archive  Electronic 
Briefing  Book  No.  13.  http://www.gwu.edu/~nsarchiv/NSAEBB/NSAEBB13/. 
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Precision  Guided  Weapons  or  Precision  Guided  Munitions 

Precision-guided  munitions  are  self-guiding  weapons  intended  to  maximize  damage  to 
the  target  while  minimizing  “collateral  damage.”  The  creation  of  precision-guided  munitions 
resulted  in  the  renaming  of  older  bombs  as  “gravity  bombs,”  “dumb  bombs,”  or  “iron  bombs.” 
Through  sustaining  innovation,  precision-guided  weapons  evolved  through  three  primary  types  - 
laser  guided,  radio-controlled,  and  satellite  guided.  The  use  of  these  weapons  has  enabled 
disruptive  innovations  to  occur,  such  as  the  CONOPS  development  leading  to  the  use  of  B-52s 
for  close  air  support  during  Operation  Enduring  Freedom  in  Afghanistan. 


Radio-Controlled.  The  United  States  Anny  (Air  Corps)  began  experimenting  with 
radio-controlled  remotely  guided  planes  in  World  War  I,  but  the  program  had  few  successes. 
The  first  successful  experiments  with  guided  bombs  were  conducted  during  World  War  II  when 
camera-guided  bombs,  flare-sighted  bombs,  and  other  steerable  munitions  were  developed.  In 
the  1960s,  the  electro-optical  bomb  or  camera  bomb  was  introduced.  They  were  equipped  with 
television  cameras  and  steerable  flare  sights,  in  which  the  bomb  would  be  steered  until  the  flare 
superimposed  the  target.  The  camera  bombs  transmitted  a  "bomb's  eye  view"  of  the  target  back 
to  a  controlling  aircraft  which  then  transmitted  control  signals  to  steerable  fins  fitted  to  the  bomb. 
Such  weapons  were  used  increasingly  by  the  USAF  in  the  last  few  years  of  the  Vietnam  War. 

Laser-Guided.  By  1967  the  USAF  had  conducted  a  competitive  evaluation  leading  to 
full  development  of  the  world's  first  laser-guided  bomb  -  the  BOLT-1 17  in  1968.  Laser-guided 
munitions  rely  on  the  target  being  illuminated  by  an  encrypted  laser  "target  designator"  on  the 
ground  or  on  an  aircraft. 

Laser-guided  weapons  did  not  become  commonplace  until  the  advent  of  the  microchip. 
The  first  large-scale  use  of  smart  weapons  came  in  1991  during  Operation  Desert  Stonn  by 
coalition  forces  against  Iraq.  Laser-guided  weapons  have  a  disadvantage  in  that  their 
effectiveness  is  impaired  during  poor  weather  conditions. 

Satellite-Guided  (GPS).  Satellite-guided  weapons,  the  most  accurate  of  the  precisions 
weapons,  use  satellite  navigation  systems  such  as  GPS  (e.g.,  JDAM  (Joint  Direct  Attack 
Munition)  and  JSOW  (Joint  Stand-Off  Weapon).  This  offers  improved  accuracy  compared  to 
laser  systems,  and  can  operate  in  all  weather  conditions,  without  any  need  for  ground  support. 
The  bomb  reverts  to  inertial  navigation  in  the  event  of  losing  the  GPS  signal. 


Ref: 

“Precision-guided  Munition.”  http://en.wikipedia.org/wiki/Precision-guided  munition. 
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Stealth 


By  the  mid-1970s,  the  Soviet-designed  and  widely  distributed  (radar-based)  Integrated 
Air  Defense  System  (IADS)  had  become  a  serious  concern  to  the  free  world.  In  conflicts  such  as 
Rolling  Thunder  in  North  Vietnam  and  the  1973  Yom  Kippur  war,  losses  were  at  times 
unsustainable,  and  the  forecast  for  losses  in  a  major  conflict  involving  NATO’s  forces  was  a 
grim  one  -  owing  in  large  measure  to  the  IADS’  effectiveness. 

It  had  been  shown  through  the  years  that  the  radar  cross  section  (RCS)  of  aircraft  could 
be  reduced  by  shaping  and  other  methods,  but  the  fact  that  radar  detection  range  for  any 
particular  aircraft  varied  as  the  fourth  root  of  its  RCS  requires  that  many  orders  of  magnitude 
reduction  over  extant  values  would  be  required  to  achieve  operational  utility.  In  1974,  Kent 
Kresa  (later  CEO  of  Northrop)  and  Ken  Perko,  both  at  DARPA,  began  a  program  to  determine 
what  levels  were  practically  achievable — and  to  build  actual  aircraft  if  possible.  They  contracted 
with  major  aircraft  companies  (originally  not  Lockheed)  for  analyses  and  computer  simulations. 
Ed  Martin,  Kelly  Johnson  (“Skunk  Works”  Director),  and  Ben  Rich  of  Lockheed  convinced 
DARPA  that  they  should  be  included.  Denys  Overholser,  Bill  Scherrer,  and  Bill  Schroeder  at 
Lockheed  came  up  with  a  “faceted”  design  approach,  and  computer  programs  which  they 
believed  predicted  that  a  very  low  observable  (VLO)  with  satisfactory  performance  could  be 
achieved1.  In  addition  to  analyses  and  computations,  other  venues  were  used  to  experimentally 
evaluate  RCS,  such  as  testing  scale  models  in  Lockheed’s  anechoic  radar  chamber. 

DARPA  awarded  contracts  to  Lockheed  and  Northrop  (the  Experimental  Survivable 
Testbed,  or  XST,  program)  in  1975  to  actually  build  full  scale  RCS  evaluation  models,  as  phase  I 
of  an  effort  to  achieve  flying  prototypes.  The  teams  in  the  contractors’  plants  were  expanded, 
and  in  March  1976  a  “pole-off’2  was  accomplished  at  the  Government’s  RCS  evaluation  facility 
(RATSCAT),  with  Lockheed  selected  to  build  the  two  flight  demonstrators.  The  program  name 
changed  to  Have  Blue,  and  was  taken  under  Air  Force  Leadership.  Key  to  the  subsequent 
success  was  Dr.  Curry  (DDR&E),  Gen.  Jones  (CSAF),  and  Lt.  Gen.  Bond.  Much  of  the  program 
was  now  similar  to  a  conventional  fighter  program: 


Preparatory >  to  and  concurrent  with  aircraft  construction,  there  were  roughly  1,000  hours  of 
high  and  low  speed  wind  tunnel  testing,  500  hours  of  propulsion  wind  tunnel  testing,  88  test 
days  at  McDonnell  Douglas  ’  Gray  Butte  RCS  range  with  a  1/3  scale  model,  and  40  days  at 
RATSCAT  with  a  full-scale  RCS  model.  Other  tests  included  structural  testing,  mockup, 
continued  flight  control  development  and  simulation,  and  “iron  bird’’  functional  mockup 
testing .3 

The  Have  Blue  demonstrators  flew  from  late  1977  through  mid  1979,  completing  88 
sorties  and,  although  both  aircraft  ultimately  were  lost,  they  accomplished  all  test  objectives  and 
demonstrated  that  tactically  useful  VLO  aircraft  were  feasible.  Ultimately  59  F-117A  aircraft 


1  A  major  part  of  the  mathematics  to  predict  RCS  was  supplied  by  a  paper  written  by  (Soviet)  Dr.  P.  I. 
Ufimtsev  in  1962. 

2  RCS  of  models  and  full-size  vehicles  is  measured  on  a  range — with  a  radar  illuminating  and  sensing  the 
returns  from  the  test  article,  mounted  on  a  pylon  (or  pole),  at  varying  aspects  of  azimuth  and  elevation. 

3  Have  Blue  and  the  F-117A  (Final  Draft),  Aronstein  and  Piccirillo,  12  March  1997  (later  published  by 
AIAA),  p.  34. 
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(essentially  a  scaled-up  Have  Blue),  costing  approximately  $6B  were  procured  under  the  Senior 
Trend  program  and  have  been  used  with  exceptional  effect  in  conflicts  around  the  world. 


Ref: 

Aronstein,  D.  and  Piccirillo,  A.  (1997).  Have  Blue  and  the  F-l  17A:  Evolution  of  the  “Stealth 
Fighter.”  Reston  VA:  American  Institute  of  Aeronautics  and  Astronautics,  Inc.  ISBN  1-56347- 
245-7. 
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Armed  Predator 

The  Armed  Predator  is  another  example  of  the  great  things  that  can  be  accomplished  in  a 
very  short  time  when  you  marry  up  the  right  vision  with  the  right  team.  The  weaponized 
Predator  was  the  brainchild  of  General  John  P.  Jumper,  who  later  became  Air  Force  Chief  of 
Staff,  when  he  was  commander  of  the  Air  Force  Air  Combat  Command.  He  tasked  Air  Force 
Acquisition  to  “demonstrate  a  weaponized  UAV”  with  the  ability  to  “find  a  target,  then  eliminate 
it.”  This  was  consistent  with  his  vision  for  greatly  reducing  the  sensor  to  shooter  kill  times. 

The  decision  was  made  to  use  the  Predator  as  the  UAV  and  the  Hellfire  Missile  as  the 
Weapon,  two  systems  that  were  designed  and  developed  without  the  other  in  mind.  The  team 
that  was  tasked  in  July  2000  to  make  Gen  Jumper’s  vision  a  reality  had  to  consider  several  risks 
inherent  in  integrating  these  two  systems:  could  the  Predator  carry  the  missile,  could  the  aircraft 
provide  the  necessary  stability  to  launch  the  missile,  and  could  the  aircraft's  onboard 
acquisition/designator  system  provide  the  stability  to  meet  the  missile's  guidance  requirements. 
All  of  this  had  to  be  done  in  a  short  timeframe. 

A  key  to  the  success  of  this  effort  was  the  fact  that  the  team  did  keep  its  focus  on  these 
three  key  technical  issues  while  also  exploring  operational  considerations,  and  it  executed  a 
campaign  of  experiments  that  led  to  a  successful  firing  of  a  live  Hellfire  missile  on  Feb  21,  2001, 
just  eight  months  after  it  got  the  tasking. 

The  Armed  Predator  project  is  another  great  example  of  the  value  of  the  key  SLE 
Principles:  A  Champion  with  a  vision  (Gen  Jumper),  building  the  right  team,  and  executing  a 
campaign  of  system  level  experiments. 


Ref: 

“Predator  Hellfire  Missile  Tests  "Totally  Successful"”  May  12,  2001.  Checkpoint. 
http://www.checkpoint-online.ch/CheckPoint/J4/J4-0003-PredatorHellfireMissileTests.html 
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